eess.IVOct 27, 2026

Image Modeling: From Handcrafted Priors to Foundation Image Models

Authors: A. Murat Tekalp

Organizations: Department of Electrical and Electronics Engineering, Koc¸ University, Istanbul, Turkey

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This paper unifies six decades of image modeling, evolving from handcrafted mathematical tractable priors to solve inverse problems in imaging to data-driven deep foundation image models for both image generation and solving inverse problems, under a common Bayesian framework that formulates image modeling as an implicit or explicit probabilistic characterization of a data manifold. Every image restoration or generative task fundamentally seeks to answer the same question: What makes an image look like a real image? In the Bayesian framework, this question is encapsulated by the prior probability density p(x), or its more optimization-friendly counterpart, the log-prior log p(x). While early works in solving inverse problems, such as Tikhonov, Total Variation (TV), and Non-local priors relied on explicit handcrafted definitions of log prior to model local smoothness, modern techniques, such as Plug-and-Play (PnP) priors and Denoising Diffusion models, are rooted in algorithmic uncoupling and implicit manifold learning that apply to both inverse problems and conditional image generation. By grounding imaging inverse problems and conditional image generation under a single mathematical engine: the gradient of the log-prior, or the Stein Score Function, we bridge the gap between signal processing and machine learning communities, turning what seems like parallel developments into a single, evolution of image modeling from handcrafted analytical models to models learned from big data, demonstrating that the pursuit of the denoising score is the common thread connecting variational optimization (in MAP estimation) to posterior sampling (in modern foundation image models).

Figures & tables

Fig. 1: The timeline of evolution of image modeling from hand-crafted priors to foundation image models. Illustration is generated using Google Gemini controlled by author prompts. The layout is designed and the content is verified by the author.
Fig. 2: Unification of image modeling from handcrafted priors to learned foundation image models based on Bayesian framework.
Fig. 3: The architecture of Foundation Image Models: Transition from Inductive Bias of CNNs to Scaling Law of transformers. (a) Traditional diffusion backbone: Convolutional U-Net, (b) The foundation model backbone: Diffusion Transformer (DiT) [ 54 ] . Illustration is generated using Google Gemini based on author prompts for layout design. The content is verified by the author.
Figure 4
More in this topic

Image Inverse Problems

Image inverse problems recover images or hidden scene properties from indirect, noisy, incomplete, or distorted measurements.

See all 119 papers

Explore similar work

CardsList
May 15, 2026

Learning Normalized Energy Models for Linear Inverse Problems

Generative diffusion models can provide powerful prior probability models for inverse problems in imaging, but existing implementations suffer from two key limitations: (i)(i) the prior density is represented implicitly, and (ii)(ii) they rely on likelihood approximations that introduce sampling biases. We address these challenges by introducing a new energy-based model trained for denoising with a covariance-based regularization term that enforces consistency across different measurement conditions. The trained model can compute normalized posterior densities for diverse linear inverse problems, without additional retraining or fine tuning. In addition to preserving the sampling capabilities of diffusion models, this enables previously unavailable capabilities: energy-guided adaptive sampling that adjusts schedules on-the-fly, unbiased Metropolis-Hastings correction steps, and blind estimation of the degradation operator via Bayes rule. We validate the method on multiple datasets (ImageNet, CelebA, AFHQ) and tasks (inpainting, deblurring), demonstrating competitive or superior performance to established baselines.

Nicolas Zilberstein, Santiago Segarra, Eero Simoncelli +1

Rice University, Houston, TX, USA · Flatiron Institute, New York, NY, USA · New York University, New York, NY, USA

Bayesian Inverse ProblemsEnergy-Based ModelsDiffusion-Based Inverse ProblemsDiffusion Posterior SamplingInverse ProblemsImage Inverse Problems
AskPDF
Apr 4, 2026

Stochastic Generative Plug-and-Play Priors

Plug-and-play (PnP) methods are widely used for solving imaging inverse problems by incorporating a denoiser into optimization algorithms. Score-based diffusion models (SBDMs) have recently demonstrated strong generative performance through a denoiser trained across a wide range of noise levels. Despite their shared reliance on denoisers, it remains unclear how to systematically use SBDMs as priors within the PnP framework without relying on reverse diffusion sampling. In this paper, we establish a score-based interpretation of PnP that justifies using pretrained SBDMs directly within PnP algorithms. Building on this connection, we introduce a stochastic generative PnP (SGPnP) framework that injects noise to better leverage the expressive generative SBDM priors, thereby improving robustness in severely ill-posed inverse problems. We provide a new theory showing that this noise injection induces optimization on a Gaussian-smoothed objective and promotes escape from strict saddle points. Experiments on challenging inverse tasks, such as multi-coil MRI reconstruction and large-mask natural image inpainting, demonstrate consistent improvement over conventional PnP methods and achieve performance competitive with diffusion-based solvers.

Chicago Y. Park, Edward P. Chandler, Yuyang Hu +4

University of Wisconsin–Madison · Washington University in St. Louis · Los Alamos National Laboratory

Image Inverse ProblemsDiffusion-Based Inverse ProblemsScore-Based Generative Modeling
AskPDF
Sep 28, 2026

Livin' on a Prior: Likelihood Score Approximation for Inverse Problems

Generative models have found great success as data-driven methods of solving inverse problems. Two popular approaches work either by combining a pretrained generative prior with a known degradation model, or by training a conditional generative model directly from paired data. We target a setting that spans both regimes: unknown degradations can be learned from few paired examples, while known degradations can be learned from self-generated samples. We introduce Likelihood Score Approximation (LSA), a generative framework that keeps a pretrained unconditional model fixed and learns an observation-conditioned model that approximates the likelihood score from paired samples. Within a conditional stochastic-interpolant framework, LSA can be trained in either score or velocity coordinates, independently of the unconditional model's native parameterization, and supports both deterministic and stochastic sampling. We further show empirically that the prior model can be swapped post-training while keeping the same LSA model. Across speech and image inverse problems, LSA operates effectively even at roughly 0.01% of the full training dataset. On the ImageNet-256 benchmark it achieves competitive or better restoration quality than strong posterior-sampling baselines while requiring up to several orders of magnitude fewer network evaluations.

Rostislav Makarov, Tal Peer, Danilo de Oliveira +1

Signal Processing University of Hamburg Hamburg, Germany

Conditional Generative ModelingInverse ProblemsImage Inverse Problems
AskPDF