A CDF-First Framework for Free-Form Density Estimation

Chenglong Song; Mazharul Islam; Lin Wang; Bing Chen; Bo Yang

cs.LGMar 26, 2026

A CDF-First Framework for Free-Form Density Estimation

Authors: Chenglong Song, Mazharul Islam, Lin Wang, Bing Chen, Bo Yang

Organizations: Shandong Key Laboratory of Ubiquitous Intelligent Computing, University of Jinan · Shandong Provincial Key Laboratory of Green and Intelligent Building Materials, University of Jinan, Jinan, Shandong 250022, China · Quancheng Shandong Laboratory, Jinan 250100, China · David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada

Abstract

Conditional density estimation (CDE) is a fundamental task in machine learning that aims to model the full conditional law $\mathbb{P}(\mathbf{y} \mid \mathbf{x})$ , beyond mere point prediction (e.g., mean, mode). A core challenge is free-form density estimation, capturing distributions that exhibit multimodality, asymmetry, or topological complexity without restrictive assumptions. However, prevailing methods typically estimate the probability density function (PDF) directly, which is mathematically ill-posed: differentiating the empirical distribution amplifies random fluctuations inherent in finite datasets, necessitating strong inductive biases that limit expressivity and fail when violated. We propose a CDF-first framework that circumvents this issue by estimating the cumulative distribution function (CDF), a stable and well-posed target, and then recovering the PDF via differentiation of the learned smooth CDF. Parameterizing the CDF with a Smooth Min-Max (SMM) network, our framework guarantees valid PDFs by construction, enables tractable approximate likelihood training, and preserves complex distributional shapes. For multivariate outputs, we use an autoregressive decomposition with SMM factors. Experiments demonstrate our approach outperforms state-of-the-art density estimators on a range of univariate and multivariate tasks.

Figures & tables

Figure 1: Comparison on toy tasks with complex support geometry: ( top ) disconnected squares and ( bottom ) ring-shaped densities. Our CDF-first model ( second column ) accurately recovers the ground-truth distribution ( leftmost column ), preserving sharp boundaries and topological holes. In contrast, MDN oversmooths multimodality, normalizing flows (NSF/MAF) fill topological voids, and DDN reduces spikiness but still blurs structural details. This demonstrates that modeling the CDF directly enables superior representation of free-form conditional densities.

Figure 2: End-to-end architecture of the CDF-first framework: input normalization, stochastic noise injection, autoregressive masked SMM networks for CDF estimation, and boundary normalization with finite-difference PDF derivation.

Dataset	Samples	$d_{x}$	$d_{y}$	Domain
Fish	908	6	1	Toxicology
Concrete	1,030	8	1	Civil Eng.
Energy	768	8	2	Building
Parkinsons	5,875	16	2	Medical
Temperature	7,588	21	2	Env.
Air	8,891	10	3	Env.

Table 1: Characteristics of UCI benchmark datasets. For each dataset, we report the number of samples, input features (

d_{x}

), and output dimensions (

d_{y}

Figure 3: Ablation study on the Elastic Ring task. Each row shows conditional density estimates at $x=-0.75,0,0.75$ for a variant of our CDF-first framework, with sum-of-squared-errors (SSE) reported per column. Only the full model (bottom row) accurately recovers the ring’s topology; removing noise, replacing SMM with hard Min-Max, or using an MLP baseline all degrade structural fidelity, demonstrating the necessity of both noise injection and smooth SMM networks for accurate density estimation.

$\delta$	5e-7	1e-6	5e-6	1e-5	3e-5
SSE	0.134	0.113	0.088	0.090	0.101

Table 2: Sensitivity of finite-difference step size

\delta

on the Elastic Ring task (SSE, lower is better).

Figure 4: Conditional density estimation on four toy tasks (Squares, Half Gaussian, Gaussian Stick, Elastic Ring) at $x=\{-0.75,-0.25,0.25,0.75\}$ . Our CDF-First model consistently recovers ground-truth structures, including disconnected modes, sharp boundaries, and topological holes, outperforming all baselines (DDN, MAF, MDN, NSF, RNF) in both visual fidelity and SSE.

	Fish	Concrete	Energy	Parkinsons	Temperature	Air	Skillcraft
MDN	0.92±0.03	0.76±0.09	-0.99±0.15	1.26±0.05	1.08±0.03	1.26±0.04	2.27±1.14
MAF	1.10±0.09	1.10±0.09	-0.13±0.29	1.54±0.08	1.25±0.04	1.14±0.08	4.06±0.28
NSF	1.57±0.19	1.32±0.29	9.23±1.63	1.72±0.13	1.79±0.16	1.50±0.17	6.72±0.64
RNF	1.24±0.08	1.24±0.09	1.89±0.20	2.30±0.07	1.84±0.10	1.70±0.11	4.49±0.19
DDN	0.92±0.04	0.59±0.03	-1.66±0.18	0.36±0.04	0.74±0.04	0.93±0.05	1.66±0.07
CDF-First	0.89±0.01	0.44±0.05	-2.03±0.19	0.73±0.04	0.70±0.06	1.06±0.03	1.65±0.08

Table 3: Test negative log-likelihood (mean ± std, lower is better) on UCI datasets with output dimensionalities: Fish (1D) Concrete (1D), Energy (2D), Parkinsons (2D), Temperature (2D), Air (3D), Skillcraft (4D). Best results per dataset in bold.

Figure 5: Reliability diagrams for CDF-First and DDN on Concrete (a,b) and Fish (c,d) datasets. CDF-First shows closer alignment to the diagonal (perfect calibration) and lower ECE values, indicating superior calibration performance.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Four synthetic tasks used for evaluation: Disconnected Squares, Half-Gaussian, Gaussian Stick, and Elastic Ring. Each task is designed to test a specific challenge in conditional density estimation—multimodality, non-Gaussianity, rotational dependence, and topological complexity, respectively.

Figure 7: Conditional density estimates on the Elastic Ring task for different finite-difference step sizes $\delta$ . Each column corresponds to a $\delta$ value, each row to a conditioning value $x$ , with SSE values averaged over $x$ . Optimal performance occurs at $\delta=5\times 10^{-6}$ .

Figure 8: Reliability diagram for Concrete dataset (10 bins). CDF-First achieves ECE = 0.0088, compared to DDN’s 0.0165, demonstrating superior calibration.

Figure 9: Reliability diagram for Fish dataset (10 bins). CDF-First achieves ECE = 0.0045, compared to DDN’s 0.0103, showing better alignment with the diagonal (perfect calibration).

Conditional Distribution Estimation

Conditional distribution estimation models the full distribution of an outcome given covariates, enabling uncertainty-aware prediction and decisions.

See all 74 papers

Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation
May 9, 2026Zhijun Zeng, Yixuan Jiang, Pipi Hu +1Energy-Based Models Density Estimation
Qiuzhen College Tsinghua University Beijing 100084, China · Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China · Yau Mathematical Sciences Center Tsinghua University, Beijing 100084, China and Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China
Symbolic Density Estimation for Discrete Distributions
May 20, 2026Ziwen Liu, Meng LiDensity Estimation
Rice University Houston, TX
Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression
Mar 27, 2026Rafael Izbicki, Pedro L. C. RodriguesConditional Distribution Estimation Tabular Foundation Models
Department of Statistics, Federal University of São Carlos (UFSCar), Brazil. · Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, France.

May 9, 2026J/K move · Enter open · S save

Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation

Zhijun Zeng, Yixuan Jiang, Pipi Hu, Zuoqiang Shi

Qiuzhen College Tsinghua University Beijing 100084, China · Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China · Yau Mathematical Sciences Center Tsinghua University, Beijing 100084, China and Yanqi Lake Beijing Institute of Mathematical Sciences and Applications Beijing 101408, China

Energy-Based Models Density Estimation

Open paper AskPDF arXiv

Density estimation is a central primitive in probabilistic modeling, yet continuous, discrete, and mixed-variable domains are often treated by separate objectives, limiting the ability to exploit a common statistical structure across data types. Continuous score-based methods rely on log-density gradients, while discrete extensions typically use concrete score whose unbounded targets become unstable near low-probability states. We introduce Constant-Target Energy Matching (CTEM), a unified energy-based framework for density estimation on general state spaces. CTEM replaces ordinary density-ratio regression with a bounded energy-difference transform and derives from it a sample-only training objective with the constant target 1. The learned scalar potential recovers log p without partition-function estimation or explicit unbounded ratio regression. Across continuous, discrete, and mixed-variable benchmarks, CTEM substantially improves density estimation over competitive baselines and yields higher-quality samples under standard sampling procedures.

Abstract

Conditional Distribution Estimation

Explore similar work

Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation

Symbolic Density Estimation for Discrete Distributions

Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression