One-Step Score-Based Density Ratio Estimation

Wei Chen; Qibin Zhao; John Paisley; Junmei Yang; Delu Zeng

stat.MLApr 12, 2026

One-Step Score-Based Density Ratio Estimation

Authors: Wei Chen, Qibin Zhao, John Paisley, Junmei Yang, Delu Zeng

Organizations: School of Mathematics, South China University of Technology, Guangzhou 510006, China · Tensor Learning Team, RIKEN Center for Advanced Intelligence Project, Tokyo 103-0027, Japan · Department of Electrical Engineering, Columbia University, New York, NY 10027, USA · School of Electronic and Information Engineering, South China University of Technology, Guangzhou 510006, China

Abstract

Density ratio estimation (DRE) is a useful tool for quantifying discrepancies between probability distributions, but existing approaches often involve a trade-off between estimation quality and computational efficiency. Classical direct DRE methods are usually efficient at inference time, yet their performance can seriously deteriorate when the discrepancy between distributions is large. In contrast, score-based DRE methods often yield more accurate estimates in such settings, but they typically require considerable repeated function evaluations and numerical integration. We propose One-step Score-based Density Ratio Estimation (OS-DRE), a partly analytic and solver-free framework designed to combine these complementary advantages. OS-DRE decomposes the time score into spatial and temporal components, representing the latter with an analytic radial basis function (RBF) frame. This formulation converts the otherwise intractable temporal integral into a closed-form weighted sum, thereby removing the need for numerical solvers and enabling DRE with only one function evaluation. We further analyze approximation conditions for the analytic frame, and establish approximation error bounds for both finitely and infinitely smooth temporal kernels, grounding the framework in existing approximation theory. Experiments across density estimation, continual Kullback-Leibler and mutual information estimation, and near out-of-distribution detection demonstrate that OS-DRE offers a favorable balance between estimation quality and inference efficiency.

Figures & tables

Figure 1: Visualization of the variance-preserving interpolating path $\{p_{t}\}_{t\in[0,1]}$ connecting a standard Gaussian ( $p_{0}$ ) to a checkerboard distribution ( $p_{1}$ ). Top: sample evolution over time. Bottom: corresponding density contours.

Figure 2: Illustrative comparison of conventional (i) to calculate $\int_{0}^{1}\partial_{t}\log p_{t}({\bm{x}})\mathrm{d}t$ directly by some traditional numerical solvers and proposed (ii) score-based DRE method (OS-DRE). The NFE of conventional methods depends on the chosen numerical solver, whereas OS-DRE requires only one single neural network evaluation.

Kernel	Generating	Smooth	Decay	Approximation	Analytic formulas
name	function $\phi(r)$		rate	theory
Matérn ( $\nu=3/2$ )	$(1+\sqrt{3}r)\exp(-\sqrt{3}r)$	Finitely	Fast	Proposition 15	Eqs. 16 and 17
Gaussian	$\exp(-r^{2})$	Infinitely	Fast	Proposition 17 (i)	Eqs. 18 and 19
Inverse Multiquadric	$(r^{2}+1)^{-1/2}$	Infinitely	Slow	Proposition 17 (ii)	Eqs. 20 and 21
Rational Quadratic	$(r^{2}+1)^{-1}$	Infinitely	Medium	Proposition 17 (iii)	Eqs. 22 and 23

Table 1: Representative analytic RBF generating functions used in OS-DRE. All kernels listed below satisfy the basic approximation conditions of the framework and admit closed-form temporal integrals and derivatives (see Appendix B ). Their explicit approximation guarantees depend on the corresponding regularity regime.

Figure 3: Comparison of density estimates on nine structured and multimodal 2D datasets. DRE- $\infty$ and D 3 RE use numerical quadrature with $\text{NFE}\in\{2,5,10\}$ , whereas OS-DRE uses partly analytic one-step inference with $\text{NFE}=1$ .

Method	NFE	POWER	GAS	HEPMASS	MINIBOONE	BSDS300
Method	NFE	NLL/Time	NLL/Time	NLL/Time	NLL/Time	NLL/Time
DRE- $\infty$	2	$0.05\scriptstyle\pm 1.84$ /0.26	$-4.37\scriptstyle\pm 1.44$ /0.18	$19.30\scriptstyle\pm 1.31$ /0.26	$41.55\scriptstyle\pm 2.07$ /0.09	$-130.68\scriptstyle\pm 4.17$ /0.47
	5	$0.35\scriptstyle\pm 0.50$ /0.47	$-3.63\scriptstyle\pm 0.78$ /0.29	$20.24\scriptstyle\pm 0.47$ /0.46	$20.90\scriptstyle\pm 0.84$ /0.10	$-83.70\scriptstyle\pm 1.35$ /1.02
	10	$0.03\scriptstyle\pm 0.17$ /0.85	$-4.34\scriptstyle\pm 0.60$ /0.45	$20.43\scriptstyle\pm 0.52$ /0.80	$20.57\scriptstyle\pm 0.93$ /0.11	$-87.65\scriptstyle\pm 2.24$ /1.93
	50	$0.25\scriptstyle\pm 0.28$ /3.50	$-4.33\scriptstyle\pm 0.71$ /1.66	$20.67\scriptstyle\pm 0.57$ /3.48	$20.97\scriptstyle\pm 0.51$ /0.19	$-90.24\scriptstyle\pm 2.14$ /9.10
	100	$0.26\scriptstyle\pm 0.21$ /6.90	$-4.32\scriptstyle\pm 0.69$ /3.16	$20.67\scriptstyle\pm 0.57$ /6.83	$21.03\scriptstyle\pm 0.36$ /0.30	$-90.31\scriptstyle\pm 1.91$ /18.12

Table 2: Density estimation performance (NLL

\downarrow

) and inference time (seconds

\downarrow

) across five tabular benchmarks. OS-DRE achieves competitive likelihoods with minimal inference cost. Best and fastest results are highlighted. Timings are measured on a single NVIDIA TITAN X GPU.

Panel of Figure 4 — Figure 4: Real-time KL divergence estimation under distribution shifts. OS-DRE ( $\mathrm{NFE}=1$ ) stably and accurately tracks the ground truth (black), whereas quadrature-based baselines ( $\mathrm{NFE}=50$ ) suffer from noticeable lag during large distributional changes.

Figure 5: MI estimation error (MSE $\downarrow$ ) on four geometrically pathological distributions ( Czyż et al., 2023 ) across varying dependency levels. OS-DRE ( $\mathrm{NFE}=1$ ) consistently yields favorable performance compared to solver-based baselines ( $\mathrm{NFE}=100$ ), demonstrating robust estimation capabilities under complex geometric topologies.

Panel of Figure 6 — Figure 6: MI estimates and MAE ( $\downarrow$ ) under the complex dependencies of Hu et al. (2024) . OS-DRE provides highly accurate and robust estimates.

Method	NFE	Kernel	MI = 10		MI = 20		MI = 30		MI = 40
Method	NFE	Kernel	MSE	Time	MSE	Time	MSE	Time	MSE	Time
DRE- $\infty$	$2$	–	$73.91$	$0.045$	$283.52$	$0.045$	$614.62$	$0.045$	$1215.69$	$0.046$
	$5$	–	$2.86$	$0.055$	$7.09$	$0.060$	$25.29$	$0.068$	$70.36$	$0.057$
	$10$	–	$0.27$	$0.075$	$0.54$	$0.100$	$2.66$	$0.085$	$7.08$	$0.083$
	$50$	–	$0.03$	$0.226$	$0.04$	$0.249$	$0.48$	$0.228$	$3.77$	$0.271$
	$100$	–	$0.02$	$0.475$	$0.01$	$0.493$	$0.50$	$0.478$	$3.31$	$0.554$

Table 3: MI estimation (MSE

\downarrow

) and inference time (seconds

\downarrow

) in high-discrepancy regimes (

\geq 20

nats) ( Rhodes et al., 2020 ) . OS-DRE successfully mitigates the density-chasm problem, delivering competitive estimation quality with low inference cost. Best and fastest results are highlighted. Timings are measured on a single NVIDIA TITAN X GPU.

Method	Near-AUROC $\uparrow$	Far-AUROC $\uparrow$	AUROC $\uparrow$	FPR@ $95$ $\downarrow$	AUPR-IN $\uparrow$	AUPR-OUT $\uparrow$	Params $\downarrow$	Time $\downarrow$
CIFAR-10
ChiSq	$0.53_{\scriptstyle\pm 0.00}$	$0.59_{\scriptstyle\pm 0.16}$	$0.56_{\scriptstyle\pm 0.08}$	$0.87_{\scriptstyle\pm 0.13}$	$0.47_{\scriptstyle\pm 0.09}$	$0.67_{\scriptstyle\pm 0.03}$	$11.2$ M	$38.4_{\scriptstyle\pm 0.6}$
Logistic	$0.53_{\scriptstyle\pm 0.02}$	$0.55_{\scriptstyle\pm 0.05}$	$0.54_{\scriptstyle\pm 0.02}$	$0.92_{\scriptstyle\pm 0.04}$	$0.41_{\scriptstyle\pm 0.02}$	$0.67_{\scriptstyle\pm 0.02}$	$11.2$ M	$38.9_{\scriptstyle\pm 1.3}$
NCE	$0.53_{\scriptstyle\pm 0.02}$	$0.55_{\scriptstyle\pm 0.05}$	$0.54_{\scriptstyle\pm 0.02}$	$0.92_{\scriptstyle\pm 0.04}$	$0.41_{\scriptstyle\pm 0.02}$	$0.67_{\scriptstyle\pm 0.02}$	$11.2$ M	$39.7_{\scriptstyle\pm 1.7}$
InfoNCE	$0.54_{\scriptstyle\pm 0.01}$	$0.65_{\scriptstyle\pm 0.06}$	$0.59_{\scriptstyle\pm 0.03}$	$0.93_{\scriptstyle\pm 0.01}$	$0.49_{\scriptstyle\pm 0.01}$	$0.69_{\scriptstyle\pm 0.01}$	$11.2$ M	$40.1_{\scriptstyle\pm 0.7}$
PW-DRE	$0.54_{\scriptstyle\pm 0.01}$	$0.56_{\scriptstyle\pm 0.10}$	$0.55_{\scriptstyle\pm 0.05}$	$0.93_{\scriptstyle\pm 0.05}$	$0.45_{\scriptstyle\pm 0.04}$	$0.67_{\scriptstyle\pm 0.03}$	$11.2$ M	$39.9_{\scriptstyle\pm 0.7}$

Table 4: OOD detection performance on CIFAR-10 and CIFAR-100 (ID). Metrics are reported as mean

\pm

std over 3 random seeds. Arrows indicate whether higher (

\uparrow

) or lower (

\downarrow

) values are better. Inference time is measured on a single NVIDIA RTX A6000 GPU.

Figure 7: Densities of OOD scores for ID CIFAR-100 (blue) vs. Near- and Far-OOD datasets. OS-DRE and D 3 RE exhibit clearer distributional discrepancies.

Panel of Figure 8 — Figure 8: Trade-off between estimation quality and computational efficiency.

Density Ratio Estimation

Density ratio estimation compares probability mass across distributions to support inference, distribution-shift correction, and data selection.

See all 26 papers

Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation
May 9, 2026Zhijun Zeng, Yixuan Jiang, Pipi Hu +1Energy-Based Models Density Estimation
Qiuzhen College Tsinghua University Beijing 100084, China · Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China · Yau Mathematical Sciences Center Tsinghua University, Beijing 100084, China and Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China
In-Context Density Estimation for Tabular Data
Aug 10, 2026Patryk Marszałek, Jacek Tabor, Marek ŚmiejaEnergy-Based Models Unsupervised Anomaly Detection
Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland
Guidance for Prior Change via Density Ratio Estimation
Aug 22, 2026Yichen Zang, Song Liu, Jiun-Yi LinInference-Time Guidance Simulation-Based Inference
Department of Mathematics University of Bristol Bristol, UK

May 9, 2026J/K move · Enter open · S save

Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation

Zhijun Zeng, Yixuan Jiang, Pipi Hu, Zuoqiang Shi

Qiuzhen College Tsinghua University Beijing 100084, China · Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China · Yau Mathematical Sciences Center Tsinghua University, Beijing 100084, China and Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China

Energy-Based Models Density Estimation

Open paper AskPDF arXiv

Density estimation is a central primitive in probabilistic modeling, yet continuous, discrete, and mixed-variable domains are often treated by separate objectives, limiting the ability to exploit a common statistical structure across data types. Continuous score-based methods rely on log-density gradients, while discrete extensions typically use concrete score whose unbounded targets become unstable near low-probability states. We introduce Constant-Target Energy Matching (CTEM), a unified energy-based framework for density estimation on general state spaces. CTEM replaces ordinary density-ratio regression with a bounded energy-difference transform and derives from it a sample-only training objective with the constant target 1. The learned scalar potential recovers log p without partition-function estimation or explicit unbounded ratio regression. Across continuous, discrete, and mixed-variable benchmarks, CTEM substantially improves density estimation over competitive baselines and yields higher-quality samples under standard sampling procedures.

Abstract

Density Ratio Estimation

Explore similar work

Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation

In-Context Density Estimation for Tabular Data

Guidance for Prior Change via Density Ratio Estimation