cs.LGApr 1, 2026

Deep Networks Favor Simple Data

Authors: Weyl Lu, Chenjie Hao, Yubei Chen

Organizations: UC Davis, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Estimated density is often interpreted as indicating how typical a sample is under a model. Yet deep models trained on one dataset can assign higher density to simpler out-of-distribution (OOD) data than to in-distribution test data. We refer to this behavior as the OOD anomaly. Prior work typically studies this phenomenon within a single architecture, detector, or benchmark, implicitly assuming certain canonical densities. We instead separate the trained network from the density estimator built from its representations or outputs. We introduce two estimators: Jacobian-based estimators and autoregressive self-estimators, making density analysis applicable to a wide range of models. Applying this perspective to a range of models, including iGPT, PixelCNN++, Glow, score-based diffusion models, DINOv2, and I-JEPA, we find the same striking regularity that goes beyond the OOD anomaly: lower-complexity samples receive higher estimated density, while higher-complexity samples receive lower estimated density. This ordering appears within a test set and across OOD pairs such as CIFAR-10 and SVHN, and remains highly consistent across independently trained models. To quantify these orderings, we introduce Spearman rank correlation and find striking agreement both across models and with external complexity metrics. Even when trained only on the lowest-density (most complex) samples - or even a single such sample - the resulting models still rank simpler images as higher density. These observations lead us beyond the original OOD anomaly to a more general conclusion: deep networks consistently favor simple data. Our goal is not to close this question, but to define and visualize it more clearly. We broaden its empirical scope and show that it appears across architectures, objectives, and density estimators.

Figures & tables

Figure 1: Density rankings on the CIFAR-10 test set. For each model, CIFAR-10 test images are sorted by estimated density (high → low) and visualized through stratified samples along this ranking. Top: base models trained on the full training set. Middle (LDT10): models retrained on the lowest-density training subset (the lowest 10% of training samples). Bottom (LDT1): models trained on a single lowest-density example. Across all settings the ranking consistently progresses from simple to complex images (even under single-sample training). All base models identify the same training image (CIFAR-10 id 29920) as the lowest-density sample; the corresponding LDT10 subset examples and the LDT1 example are shown on the right bottom.
Figure 2: Spearman correlations between rankings induced by the base models and by external complexity proxies. The lower triangle is shown. Positive values mean two methods rank CIFAR-10 images in a similar rank from simple / high-density to complex / low-density. The row / column labeled “Glow+JPEG complexity” is the JPEG-based correction inspired by Serra et al. [ 17 ] .
Figure 3: Spearman correlations between rankings induced by different training settings within each architecture family. Base uses the full CIFAR-10 training set, LDT10 uses only the lowest-density 10% of the training set, LDT1 uses only the single lowest-density image, and UT is the untrained network. Each LDT setting is shown at two widely separated checkpoints to reduce the chance of reporting a transient undertrained state.
Figure 4: PixelCNN++ after training on a single lowest-density image. Unlike iGPT and Glow, the single-sample PixelCNN++ ranking no longer follows the global simplicity ranking from the base model. The failure is already visible quantitatively in the middle panel of Fig. 3 and is shown qualitatively here.
Figure 5: DINOv2 on bicubically upsampled CIFAR-10. Compared with the other models, the density bins display a much weaker monotone progression from simple to complex. This is consistent with the weak DINOv2 row / column in Fig. 2 .
Figure 6: Density rankings on the ImageNet eagle subset. Left: qualitative rankings produced by Diffusion, I-JEPA, and DINOv2 show a consistent simple-to-complex ranking of images. Right: Spearman rank correlations confirm substantial agreement between the rankings produced by different models.
Figure 7
More in this topic

Density Estimation

Density estimation learns probability distributions from data to measure likelihood, detect anomalies, quantify uncertainty, and generate samples.

See all 66 papers

Explore similar work

CardsList
Aug 10, 2026

In-Context Density Estimation for Tabular Data

Density estimation underlies many unsupervised tasks on tabular data such as anomaly detection, out-of-distribution detection, and data augmentation. Although all these problems reduce to questions about where probability mass lies, they are typically solved individually by fitting a separate model to each dataset, with its own hyperparameters and tuning budget. We introduce ICED, an in-context, energy-based density estimator that removes this per-dataset cost. ICED is a transformer-based model pretrained once on a synthetic prior built specifically for density estimation under an objective that fits log-density where it is informative and preserves its ordering elsewhere. In the inference, it reads a dataset as context and returns an unnormalized log-density for any query point in a single forward pass, with no fitting, sampling, or hyperparameter selection. A single frozen ICED model then drives four tasks usually handled by four specialized pipelines: density estimation, out-of-distribution detection, unsupervised anomaly detection, and generative augmentation. Across all four, it is competitive with the strongest task-specific method, while being the only approach that needs no retraining, no tuning, and no labels to move between them. The code is available at https://github.com/gmum/iced.

Patryk Marszałek, Jacek Tabor, Marek Śmieja

Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland

Energy-Based ModelsUnsupervised Anomaly DetectionOOD DetectionTabular ICLDensity Estimation
AskPDF
Apr 6, 2026

Face Density as a Proxy for Data Complexity: Quantifying the Hardness of Instance Count

Machine learning progress has historically prioritized model-centric innovations, yet achievable performance is frequently capped by the intrinsic complexity of the data itself. In this work, we isolate and quantify the impact of instance density (measured by face count) as a primary driver of data complexity. Rather than simply observing that ``crowded scenes are harder,'' we rigorously control for class imbalance to measure the precise degradation caused by density alone. Controlled experiments on the WIDER FACE and Open Images datasets, restricted to exactly 1 to 18 faces per image with perfectly balanced sampling, reveal that model performance degrades monotonically with increasing face count. This trend holds across classification, regression, and detection paradigms, even when models are fully exposed to the entire density range. Furthermore, we demonstrate that models trained on low-density regimes fail to generalize to higher densities, exhibiting a systematic under-counting bias, with error rates increasing by up to 4.6x, which suggests density acts as a domain shift. These findings establish instance density as an intrinsic, quantifiable dimension of data hardness and motivate specific interventions in curriculum learning and density-stratified evaluation.

Abolfazl Mohammadi-Seif, Ricardo Baeza-Yates

Department of Engineering, Universitat Pompeu Fabra, Barcelona, Spain.

Topic pending
AskPDF
Feb 10, 2026

Mitigating the Likelihood Paradox in Flow-based OOD Detection: An Entropic Perspective

Deep generative models that can tractably compute input likelihoods, including normalizing flows, often assign unexpectedly high likelihoods to out-of-distribution (OOD) inputs. We mitigate this paradox from an entropic perspective with Semantic Proportional Entropy Manipulation (SPEM), a training-free procedure that applies stronger perturbations to inputs less similar to an in-distribution (ID) memory bank. We derive theoretical lower bounds showing that this entropy gap increases the expected log-likelihood difference in favor of the ID, and experiments on standard benchmarks show that this manipulation realigns the likelihood ordering with human intuition. Strikingly, this realignment persists even when the flow is given only visually meaningless noise instead of the original input, and we analyze this phenomenon theoretically to characterize the conditions under which it occurs.

Donghwan Kim, Hyunsoo Yoon

Department of Industrial Engineering University of Yonsei

OOD DetectionNormalizing Flows
AskPDF