cs.LGSep 25, 2026

Representation Learning for Exact Preimages

Authors: Konstantin Hess, Stefan Feuerriegel

Organizations: LMU Munich · Munich Center for Machine Learning

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Modern neural predictors can model highly nonlinear maps, but many scientific and engineering tasks require reasoning in the opposite direction: given a performance or safety level, the goal is to characterize the preimage, that is, the complete set of inputs which meet the desired target level and optimize over that set. For expressive neural predictors, however, such preimages typically have no explicit representation and are expensive to recover or optimize over. This creates a fundamental three-way challenge between expressive forward prediction, accurate preimage approximation, and tractable optimization over the preimage for downstream tasks. We introduce TRIO (tractable representations for preimage learning and inverse optimization), a framework for learning representations that make these objectives compatible by construction. Our key contribution is a preimage factorization: the forward model remains expressive through nonlinear radial transformations (including neural networks), while, under inversion, each transformation reduces to a single scalar radius, which yields simple geometric level sets. This yields an explicit geometric representation that is reusable for downstream optimization over the preimage, and, for linear objectives, we show that this admits a closed-form global solution. We finally prove a universal approximation theorem which shows that TRIO can approximate any continuous forward map and its entire family of potentially disconnected, nonconvex preimages arbitrarily well. Hence, TRIO combines expressive forward modeling, exact preimage recovery, and tractable global downstream optimization over preimages by design.

Figures & tables

Figure 1: Filling the gap: TRIO combines forward expressiveness with exact preimage approximation and tractable inverse optimization.
Figure 2: Computing the preimage: TRIO allows for non-injective, non-convex mappings and disconnected preimages.
Figure 3: TRIO : The forward predictor is learned from data; for a chosen target, the preimage can be obtained excatly, which enables downstream optimziation over independent subproblems.
Figure 4: Preimage reconstruction. For fixed neural backbone, larger QQ yields increasingly accurate geometric reconstruction, while the number of active experts varies across target levels.
Figure 5: Forward RMSE: For increasing parameters, TRIO converges quickly to the forward RMSE of the MLP, while the ICNN completely deteriorates.
Figure 6: Downstream optimization. (a) Regret wrt. objective ( ↓\downarrow lower is better); (b) Runtime per projection query ( ↓\downarrow lower is better); and (c) Constraint violation ( ↓\downarrow lower is better).
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Complex power induces highly multimodel, disconnected preimages. Shown are the preimages at different target level quantiles. TRIO can approximate these arbitrarily close.
More in this topic

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 773 papers

Explore similar work

CardsList
Aug 12, 2026

Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks

Neural networks can often be trained or fine-tuned through random low-dimensional reparameterization, where a small latent vector is mapped into a full parameter update by a frozen random map. This raises a practical question: how large must the latent search space be to reach a low-loss region? We first express the known accessibility transition in an equivalent conic form, centered for compact convex targets at the statistical dimension of the polar cone. Our main theoretical contribution is an orientation-resolved quadratic master formula that predicts the random-slice residual from both the curvature spectrum and the reference-to-solution displacement profile. It yields a self-consistent isotropic-orientation predictor and, in a conservative radius-only specialization, recovers the earlier Gaussian-width quadratic bound. Building on this analysis, we introduce Random Mapping Networks (RaMaN), which instantiate the predicted latent dimension using structured Hadamard or seed-regenerated Gaussian maps. These constructions avoid the O(dP) storage of dense random maps and reduce optimizer-state memory from O(P) to O(d). We also develop matrix-free curvature approximations and sweep-free dimension selection. Across controlled quadratic and neural-curvature experiments, the orientation-resolved predictor closely tracks measured transition locations and outperforms orientation-agnostic approximations when displacement direction matters. End-to-end experiments further show sharp, protocol-dependent training transitions across image and language models.

Andrew Cheng, Ali Eslamian, Jie Cheng +2

Department of Computer Science, Tsinghua University, Beijing, China · Department of Computer Science, The University of Manchester, Manchester, UK · Department of Computer Science, University of Kentucky, Lexington, KY, USA +3

Subspace Optimization
AskPDF
May 13, 2026

Twincher: Bijective Representation Learning for Robust Inversion of Continuous Systems

Recent advances in AI have been primarily driven by large-scale neural architectures that excel at function approximation, rather than by tailored inductive biases and inference or learning strategies that could be important for resource-efficient real-world perception and planning through the solution of inverse problems. In this work, we consider the possibility of enabling robust inversion of continuous forward processes p↦yp \mapsto y by learning representations of yy that are bijectively aligned with pp while remaining insensitive to perturbations in yy caused by noise or model mismatch. We propose Twincher, a class of architectures based on stacks of structured diffeomorphic transformations and tailored adversarial training strategies that enable learning such bijective representations. We provide a public API for training and inference and empirically demonstrate the ability of the proposed architecture to efficiently learn bijective representations of synthetic systems, thereby enabling robust and efficient iterative inverse inference. Compared to a baseline inverse-modeling approach, the method exhibits improved data efficiency and robustness, providing initial evidence for the potential of bijective representation learning in robotics, vision, and physical AI.

Arkady Gonoskov

Department of Physics, University of Gothenburg, SE-41296 Gothenburg, Sweden

Representation LearningInvariant Representation LearningInverse Problems
AskPDF
Mar 30, 2026

The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations

The approximation and convergence properties of implicit neural representations (INRs) are known to be highly sensitive to parameter initialization strategies. While several data-driven initialization methods demonstrate significant improvements over standard random sampling, the reasons for their success -- specifically, whether they encode classical statistical signal priors or more complex features -- remain poorly understood. In this study, we explore this phenomenon through a series of experimental analyses leveraging noise pretraining. We pretrain INRs on diverse noise classes (e.g., Gaussian, Dead Leaves, Spectral) and measure their ability to both fit unseen signals and encode priors for an inverse imaging task (denoising). Our analyses on image and video data reveal a surprising finding: simply pretraining on unstructured noise (Uniform, Gaussian) dramatically improves signal fitting capacity compared to all other baselines. However, unstructured noise also yields poor deep image priors for denoising. In contrast, we also find that noise with the classic 1/∣fα∣1/|f^α| spectral structure of natural images achieves an excellent balance of signal fitting and inverse imaging capabilities, performing on par with the best data-driven initialization methods. This finding enables more efficient INR training in applications lacking sufficient prior domain-specific data. For more details, visit project page at https://kushalvyas.github.io/noisepretraining.html

Kushal Vyas, Alper Kayabasi, Daniel Kim +3

Rice University · University of California, Riverside

Implicit Neural RepresentationsNeural Network InitializationSynthetic Data PretrainingSelf-Supervised Pre-Training
AskPDF