One-Step Score-Based Density Ratio Estimation

Wei Chen; Qibin Zhao; John Paisley; Junmei Yang; Delu Zeng

stat.MLApr 12, 2026

One-Step Score-Based Density Ratio Estimation

Authors: Wei Chen, Qibin Zhao, John Paisley, Junmei Yang, Delu Zeng

Organizations: School of Mathematics, South China University of Technology, Guangzhou 510006, China · Tensor Learning Team, RIKEN Center for Advanced Intelligence Project, Tokyo 103-0027, Japan · Department of Electrical Engineering, Columbia University, New York, NY 10027, USA · School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510006, China

Abstract

Density ratio estimation (DRE) is a useful tool for quantifying discrepancies between probability distributions, but existing approaches often involve a trade-off between estimation quality and computational efficiency. Classical direct DRE methods are usually efficient at inference time, yet their performance can seriously deteriorate when the discrepancy between distributions is large. In contrast, score-based DRE methods often yield more accurate estimates in such settings, but they typically require considerable repeated function evaluations and numerical integration. We propose One-step Score-based Density Ratio Estimation (OS-DRE), a partly analytic and solver-free framework designed to combine these complementary advantages. OS-DRE decomposes the time score into spatial and temporal components, representing the latter with an analytic radial basis function (RBF) frame. This formulation converts the otherwise intractable temporal integral into a closed-form weighted sum, thereby removing the need for numerical solvers and enabling DRE with only one function evaluation. We further analyze approximation conditions for the analytic frame, and establish approximation error bounds for both finitely and infinitely smooth temporal kernels, grounding the framework in existing approximation theory. Experiments across density estimation, continual Kullback-Leibler and mutual information estimation, and near out-of-distribution detection demonstrate that OS-DRE offers a favorable balance between estimation quality and inference efficiency.

Figures & tables

Figure 1: Visualization of the variance-preserving interpolating path $\{p_{t}\}_{t\in[0,1]}$ connecting a standard Gaussian ( $p_{0}$ ) to a checkerboard distribution ( $p_{1}$ ). Top: sample evolution over time. Bottom: corresponding density contours.

Figure 2: Illustrative comparison of conventional (i) to calculate $\int_{0}^{1}\partial_{t}\log p_{t}({\bm{x}})\mathrm{d}t$ directly by some traditional numerical solvers and proposed (ii) score-based DRE method (OS-DRE). The NFE of conventional methods depends on the chosen numerical solver, whereas OS-DRE requires only one single neural network evaluation.

Kernel	Generating	Smooth	Decay	Approximation	Analytic formulas
name	function $\phi(r)$		rate	theory
Matérn ( $\nu=3/2$ )	$(1+\sqrt{3}r)\exp(-\sqrt{3}r)$	Finitely	Fast	Proposition 15	Eqs. 16 and 17
Gaussian	$\exp(-r^{2})$	Infinitely	Fast	Proposition 17 (i)	Eqs. 18 and 19
Inverse Multiquadric	$(r^{2}+1)^{-1/2}$	Infinitely	Slow	Proposition 17 (ii)	Eqs. 20 and 21
Rational Quadratic	$(r^{2}+1)^{-1}$	Infinitely	Medium	Proposition 17 (iii)	Eqs. 22 and 23

Table 1: Representative analytic RBF generating functions used in OS-DRE. All kernels listed below satisfy the basic approximation conditions of the framework and admit closed-form temporal integrals and derivatives (see Appendix B ). Their explicit approximation guarantees depend on the corresponding regularity regime.

Figure 3: Comparison of density estimates on nine structured and multimodal 2D datasets. DRE- $\infty$ and D 3 RE use numerical quadrature with $\text{NFE}\in\{2,5,10\}$ , whereas OS-DRE uses partly analytic one-step inference with $\text{NFE}=1$ .

Method	NFE	POWER	GAS	HEPMASS	MINIBOONE	BSDS300
Method	NFE	NLL/Time	NLL/Time	NLL/Time	NLL/Time	NLL/Time
DRE- $\infty$	2	$0.05\scriptstyle\pm 1.84$ /0.26	$-4.37\scriptstyle\pm 1.44$ /0.18	$19.30\scriptstyle\pm 1.31$ /0.26	$41.55\scriptstyle\pm 2.07$ /0.09	$-130.68\scriptstyle\pm 4.17$ /0.47
	5	$0.35\scriptstyle\pm 0.50$ /0.47	$-3.63\scriptstyle\pm 0.78$ /0.29	$20.24\scriptstyle\pm 0.47$ /0.46	$20.90\scriptstyle\pm 0.84$ /0.10	$-83.70\scriptstyle\pm 1.35$ /1.02
	10	$0.03\scriptstyle\pm 0.17$ /0.85	$-4.34\scriptstyle\pm 0.60$ /0.45	$20.43\scriptstyle\pm 0.52$ /0.80	$20.57\scriptstyle\pm 0.93$ /0.11	$-87.65\scriptstyle\pm 2.24$ /1.93
	50	$0.25\scriptstyle\pm 0.28$ /3.50	$-4.33\scriptstyle\pm 0.71$ /1.66	$20.67\scriptstyle\pm 0.57$ /3.48	$20.97\scriptstyle\pm 0.51$ /0.19	$-90.24\scriptstyle\pm 2.14$ /9.10
	100	$0.26\scriptstyle\pm 0.21$ /6.90	$-4.32\scriptstyle\pm 0.69$ /3.16	$20.67\scriptstyle\pm 0.57$ /6.83	$21.03\scriptstyle\pm 0.36$ /0.30	$-90.31\scriptstyle\pm 1.91$ /18.12

Table 2: Density estimation performance (NLL

\downarrow

) and inference time (seconds

\downarrow

) across five tabular benchmarks. OS-DRE achieves competitive likelihoods with minimal inference cost. Best and fastest results are highlighted. Timings are measured on a single NVIDIA TITAN X GPU.

Panel of Figure 4 — Figure 4: Real-time KL divergence estimation under distribution shifts. OS-DRE ( $\mathrm{NFE}=1$ ) stably and accurately tracks the ground truth (black), whereas quadrature-based baselines ( $\mathrm{NFE}=50$ ) suffer from noticeable lag during large distributional changes.

Figure 5: MI estimation error (MSE $\downarrow$ ) on four geometrically pathological distributions ( Czyż et al., 2023 ) across varying dependency levels. OS-DRE ( $\mathrm{NFE}=1$ ) consistently yields favorable performance compared to solver-based baselines ( $\mathrm{NFE}=100$ ), demonstrating robust estimation capabilities under complex geometric topologies.

Panel of Figure 6 — Figure 6: MI estimates and MAE ( $\downarrow$ ) under the complex dependencies of Hu et al. (2024) . OS-DRE provides highly accurate and robust estimates.

Method	NFE	Kernel	MI = 10		MI = 20		MI = 30		MI = 40
Method	NFE	Kernel	MSE	Time	MSE	Time	MSE	Time	MSE	Time
DRE- $\infty$	$2$	–	$73.91$	$0.045$	$283.52$	$0.045$	$614.62$	$0.045$	$1215.69$	$0.046$
	$5$	–	$2.86$	$0.055$	$7.09$	$0.060$	$25.29$	$0.068$	$70.36$	$0.057$
	$10$	–	$0.27$	$0.075$	$0.54$	$0.100$	$2.66$	$0.085$	$7.08$	$0.083$
	$50$	–	$0.03$	$0.226$	$0.04$	$0.249$	$0.48$	$0.228$	$3.77$	$0.271$
	$100$	–	$0.02$	$0.475$	$0.01$	$0.493$	$0.50$	$0.478$	$3.31$	$0.554$

Table 3: MI estimation (MSE

\downarrow

) and inference time (seconds

\downarrow

) in high-discrepancy regimes (

\geq 20

nats) ( Rhodes et al., 2020 ) . OS-DRE successfully mitigates the density-chasm problem, delivering competitive estimation quality with low inference cost. Best and fastest results are highlighted. Timings are measured on a single NVIDIA TITAN X GPU.

Method	Near-AUROC $\uparrow$	Far-AUROC $\uparrow$	AUROC $\uparrow$	FPR@ $95$ $\downarrow$	AUPR-IN $\uparrow$	AUPR-OUT $\uparrow$	Params $\downarrow$	Time $\downarrow$
CIFAR-10
ChiSq	$0.53_{\scriptstyle\pm 0.00}$	$0.59_{\scriptstyle\pm 0.16}$	$0.56_{\scriptstyle\pm 0.08}$	$0.87_{\scriptstyle\pm 0.13}$	$0.47_{\scriptstyle\pm 0.09}$	$0.67_{\scriptstyle\pm 0.03}$	$11.2$ M	$38.4_{\scriptstyle\pm 0.6}$
Logistic	$0.53_{\scriptstyle\pm 0.02}$	$0.55_{\scriptstyle\pm 0.05}$	$0.54_{\scriptstyle\pm 0.02}$	$0.92_{\scriptstyle\pm 0.04}$	$0.41_{\scriptstyle\pm 0.02}$	$0.67_{\scriptstyle\pm 0.02}$	$11.2$ M	$38.9_{\scriptstyle\pm 1.3}$
NCE	$0.53_{\scriptstyle\pm 0.02}$	$0.55_{\scriptstyle\pm 0.05}$	$0.54_{\scriptstyle\pm 0.02}$	$0.92_{\scriptstyle\pm 0.04}$	$0.41_{\scriptstyle\pm 0.02}$	$0.67_{\scriptstyle\pm 0.02}$	$11.2$ M	$39.7_{\scriptstyle\pm 1.7}$
InfoNCE	$0.54_{\scriptstyle\pm 0.01}$	$0.65_{\scriptstyle\pm 0.06}$	$0.59_{\scriptstyle\pm 0.03}$	$0.93_{\scriptstyle\pm 0.01}$	$0.49_{\scriptstyle\pm 0.01}$	$0.69_{\scriptstyle\pm 0.01}$	$11.2$ M	$40.1_{\scriptstyle\pm 0.7}$
PW-DRE	$0.54_{\scriptstyle\pm 0.01}$	$0.56_{\scriptstyle\pm 0.10}$	$0.55_{\scriptstyle\pm 0.05}$	$0.93_{\scriptstyle\pm 0.05}$	$0.45_{\scriptstyle\pm 0.04}$	$0.67_{\scriptstyle\pm 0.03}$	$11.2$ M	$39.9_{\scriptstyle\pm 0.7}$

Table 4: OOD detection performance on CIFAR-10 and CIFAR-100 (ID). Metrics are reported as mean

\pm

std over 3 random seeds. Arrows indicate whether higher (

\uparrow

) or lower (

\downarrow

) values are better. Inference time is measured on a single NVIDIA RTX A6000 GPU.

Figure 7: Densities of OOD scores for ID CIFAR-100 (blue) vs. Near- and Far-OOD datasets. OS-DRE and D 3 RE exhibit clearer distributional discrepancies.

Panel of Figure 8 — Figure 8: Trade-off between estimation quality and computational efficiency.

Density Ratio Estimation

Density ratio estimation compares probability mass across distributions to support inference, distribution-shift correction, and data selection.

See all 26 papers

May 9, 2026

Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation

Density estimation is a central primitive in probabilistic modeling, yet continuous, discrete, and mixed-variable domains are often treated by separate objectives, limiting the ability to exploit a common statistical structure across data types. Continuous score-based methods rely on log-density gradients, while discrete extensions typically use concrete score whose unbounded targets become unstable near low-probability states. We introduce Constant-Target Energy Matching (CTEM), a unified energy-based framework for density estimation on general state spaces. CTEM replaces ordinary density-ratio regression with a bounded energy-difference transform and derives from it a sample-only training objective with the constant target 1. The learned scalar potential recovers log p without partition-function estimation or explicit unbounded ratio regression. Across continuous, discrete, and mixed-variable benchmarks, CTEM substantially improves density estimation over competitive baselines and yields higher-quality samples under standard sampling procedures.

Zhijun Zeng, Yixuan Jiang, Pipi Hu +1

Qiuzhen College Tsinghua University Beijing 100084, China · Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China · Yau Mathematical Sciences Center Tsinghua University, Beijing 100084, China and Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China

Energy-Based Models Density Estimation

Ask PDF

Aug 10, 2026

In-Context Density Estimation for Tabular Data

Density estimation underlies many unsupervised tasks on tabular data such as anomaly detection, out-of-distribution detection, and data augmentation. Although all these problems reduce to questions about where probability mass lies, they are typically solved individually by fitting a separate model to each dataset, with its own hyperparameters and tuning budget. We introduce ICED, an in-context, energy-based density estimator that removes this per-dataset cost. ICED is a transformer-based model pretrained once on a synthetic prior built specifically for density estimation under an objective that fits log-density where it is informative and preserves its ordering elsewhere. In the inference, it reads a dataset as context and returns an unnormalized log-density for any query point in a single forward pass, with no fitting, sampling, or hyperparameter selection. A single frozen ICED model then drives four tasks usually handled by four specialized pipelines: density estimation, out-of-distribution detection, unsupervised anomaly detection, and generative augmentation. Across all four, it is competitive with the strongest task-specific method, while being the only approach that needs no retraining, no tuning, and no labels to move between them. The code is available at https://github.com/gmum/iced.

Patryk Marszałek, Jacek Tabor, Marek Śmieja

Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland

Energy-Based Models Unsupervised Anomaly Detection OOD Detection Tabular ICL Density Estimation

Ask PDF

Aug 22, 2026

Guidance for Prior Change via Density Ratio Estimation

Simulation-Based Inference (SBI) serves as a vital framework for parameter inference in scientific fields where simulators involve intractable likelihoods, yet while amortized generative models offer rapid posterior estimation, they are often restricted by the specific priors used during training, thereby limiting their flexibility as prior knowledge evolves. To address this prior dependency, PriorGuide was introduced as an inference-time guidance method, but due to its intractable formulation, it relies on Gaussian approximations of the reverse transition kernel and Gaussian mixture model fitting for the prior ratio, both of which introduce systematic bias. Motivated by these limitations, we propose an unbiased test-time guidance framework that leverages Density Ratio Estimation (DRE) to learn a score guidance term, effectively decoupling the inference process from the prior training. Moreover, our framework remains agnostic to the specific density ratio estimators, making it a general and flexible framework for handling prior changes. Experimental results across multiple tasks demonstrate that our method matches or outperforms PriorGuide on C2ST and MMD in most tasks while maintaining robustness even under limited overlap between the training and target priors. Furthermore, we apply our method to Bayesian updating for parameter inference from planetary light-curve data, where it also demonstrates strong effectiveness and robustness. Code is available at https://github.com/a-chenchen/dre-based-prior-guidance .

Yichen Zang, Song Liu, Jiun-Yi Lin

Department of Mathematics University of Bristol Bristol, UK

Inference-Time Guidance Simulation-Based Inference Bayesian Inference

Ask PDF

Abstract

Density Ratio Estimation

Explore similar work

Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation

In-Context Density Estimation for Tabular Data

Guidance for Prior Change via Density Ratio Estimation