Amortized Set Prediction for Inverse IFS Reconstruction from Density Maps
Authors: Yutaka Yamaguti
Organizations: Faculty of Information Engineering, Fukuoka Institute of Technology, 3-30-1 Wajiro-higashi, Higashi-ku, Fukuoka, 811-0295, Fukuoka, Japan
Iterated Function Systems (IFS) generate self-similar fractals from a few contractive affine maps. The forward map from parameters to images is computationally inexpensive and well understood, whereas the inverse problem of estimating maps from an image is difficult and is typically handled by per-image optimization. We replace this loop with a single forward pass of a learned estimator that predicts the affine-map set directly from a visit-frequency density map, thereby amortizing the inverse problem. The design follows two constraints. First, density maps do not uniquely identify IFS parameters, so evaluation is based on reconstruction rather than parameter recovery; unordered map sets are handled by Hungarian matching, and ground-truth parameters provide a stable training surrogate. Second, the fully known forward model lets us generate exact synthetic training pairs and also supports image-only test-time refinement. On in-distribution tests, amortized initialization plus a few refinement steps lies on a better quality--speed frontier than equal-budget random-initialized per-image optimization, and a 30-step refinement (about 0.56 s per sample) remains better than a doubled-budget baseline. Extending optimization to 1000 steps shows that the benefit is not only speed: amortized initialization reaches high-quality reconstructions more frequently than random starts. On real images (MNIST and Fashion-MNIST), it improves density metrics on average over a published per-image optimizer while being roughly 12 to 2600 times faster.
Figures & tables
Figure 1: Identifiability (all pairs of test256 ). Left: density distance versus parameter-set distance (Pearson correlation −0.146 ). The 59 near-image, far-parameter pairs that simultaneously satisfy the thresholds (density ≤0.0168 , the bottom 1% ; parameter distance ≥1.299 , the top 10% ) are shown in red. Right: parameter-set distance (median/p90/max) among image pairs with density distance ≤ε ; it does not fall off as ε shrinks.
Distance
Mean
Median
1%
Max
Density L2
0.0340
0.0325
0.0168
0.0778
(W,b) distance
1.0606
1.0559
0.6535
1.9016
Table 1: Pairwise distance distribution on test256 (density distance and parameter-set distance).
Setting
sec/sample
density SSE
Chamfer
HD95
coverage@2px
(W,b) err
base 0-step
0.0095
7.85×10−4
0.0547
0.2106
0.6650
0.405
base +10
0.192
3.85×10−4
0.0365
0.1381
0.7855
0.395
base +20
0.370
2.76×10−4
0.0295
0.1059
0.8319
0.394
base +30
0.556
2.23×10−4
0.0259
0.0913
0.8621
0.395
random-r4 0-step
0.0110
1.005×10−3
0.1546
0.5719
0.4063
0.981
random-r4 +10
0.199
7.69×10−4
0.1184
0.4566
0.5102
0.973
Table 2: Quality–speed Pareto ( test256 ). base amortized(+refinement) versus equal-budget random-from-scratch (r4). Time is seconds per sample; density SSE, Chamfer, and HD95 are better when smaller and coverage@2px when larger. The (W,b) error is the Hungarian-matched parameter-set distance (same definition as Section 5 ), reported as a diagnostic.
Figure 2: Quality–speed Pareto ( test256 ). Horizontal axis: seconds per sample. Left: Chamfer (smaller is better); right: coverage@2px (larger is better). The base amortized(+refinement) curve consistently improves over equal-budget random-from-scratch.
Figure 3: Qualitative examples of the main result ( 4 representative examples of test256 , density maps by the matched renderer). Columns: target, ours- 0 (one-shot), ours- 30 ( +30 refinement), and random-r4- 30 (equal-budget per-instance); C in each cell is the Chamfer distance. At equal budget, ours- 30 is closer to the target than random-r4- 30 in most examples (point-cloud version in Figure A.1 ).
Method
step
density SSE
CD mean
CD median
CD p95
success τ
(W,b) err
base
30
2.27×10−4
0.0262
0.0234
0.0457
0.598
0.395
base
1000
1.55×10−4
0.0195
0.0181
0.0340
0.824
0.453
random-r1
1000
2.82×10−4
0.0331
0.0314
0.0540
0.301
0.971
random-r4
1000
2.09×10−4
0.0276
0.0273
0.0417
0.445
0.928
Table 3: Convergence distribution under long optimization ( test256 , matched renderer). Converged values at 1000 steps for base (amortized initialization) and random initialization (r1 / best of r4). The CD columns are the mean, median, and p95 of Chamfer; density SSE is the mean (all better when smaller). The success rate is Pr[CD≤τ] with τ=0.0262 (mean Chamfer of base +30).
Figure 4: Convergence under long optimization ( test256 ). Left: success rate Pr[CD≤τ] ( τ=0.0262 ) versus number of steps. base (amortized initialization) reaches 0.77 by 100 steps and saturates at about 0.82 , whereas random initialization remains at 0.30 (r1) / 0.44 (r4) even at 1000 steps. Right: pairwise scatter of the 1000 -step converged Chamfer for the same samples (horizontal base , vertical random). Almost all points lie above the diagonal, so amortized initialization converges to a better solution sample by sample.
Method
step
density SSE
Chamfer
HD95
coverage@2px
(W,b) err
base
0
7.85×10−4
0.0547
0.2106
0.6650
0.405
match-only
0
8.30×10−4
0.0552
0.2121
0.6629
0.407
+aux
0
7.75×10−4
0.0527
0.2010
0.6680
0.413
base
30
2.23×10−4
0.0259
0.0913
0.8621
0.395
match-only
30
2.35×10−4
0.0263
0.0934
0.8591
0.396
+aux
30
2.26×10−4
0.0264
0.0942
0.8589
0.402
Table 4: Effect of the reconstruction auxiliary loss ( test256 ). base (no auxiliary), match-only (continuation only), and +aux (with reconstruction auxiliary) at 0 and 30 steps.
Figure 5: Quality–speed of within-family OOD (Chamfer vs seconds per sample). Left: T1 (wide translation); right: S1 (large singular values).
Method
occupancy MSE
density SSE
Chamfer
HD95
coverage@2px
ours 0 -step
0.0874
4.30×10−4
0.0571
0.2245
0.6888
ours 30 -step
0.0867
1.01×10−4
0.0221
0.0753
0.9126
ours 100 -step
0.0724
6.95×10−5
0.0193
0.0598
0.9357
ours-init + occupancy-GD 100
0.0190
7.67×10−4
0.0485
0.1931
0.7114
Tu ( n=4 )
0.0296
2.88×10−3
0.1329
0.5305
0.6154
Tu ( n=10 )
0.0214
4.91×10−3
0.1017
0.3837
0.6364
Table 6: MNIST (balanced 50 images, n=10 estimator). occupancy MSE is the minimum MSE under the occupancy condition; the rest are density metrics. density SSE, Chamfer, HD95, and occupancy MSE are better when smaller, coverage@2px when larger.
Figure 6: MNIST reconstruction (density maps, common grayscale, one example per row). Columns from left: target, ours 0 -step, ours 30 -step, ours 100 -step, ours-init + occupancy-GD 100 , Tu ( n=10 ). Under density-aware refinement, mass spreads as an area and approaches the target’s thick density as 0→30→100 steps, whereas Tu and occupancy-GD have clear outlines but concentrate mass on thin lines and isolated points.
Method
occupancy MSE
density SSE
Chamfer
HD95
coverage@2px
ours 0 -step
0.1087
1.38×10−4
0.0503
0.2102
0.7724
ours 30 -step
0.0997
3.98×10−5
0.0251
0.0738
0.8846
ours 100 -step
0.0966
3.09×10−5
0.0235
0.0652
0.8979
ours-init + occupancy-GD 100
0.0450
4.94×10−4
0.0629
0.2554
0.7645
Tu ( n=10 , 1000 iters)
0.0672
1.43×10−4
0.0453
0.1754
0.8610
Table 7: Fashion-MNIST (balanced 50 images, n=10 ). occupancy MSE is the minimum MSE under the occupancy condition; the rest are density metrics.
Figure 7: Fashion-MNIST reconstruction (density maps, common grayscale, one example per row). Columns from left: target, ours 0 -step, ours 30 -step, ours 100 -step, ours-init + occupancy-GD 100 , Tu ( n=10 ). Faithful reconstruction of filled regions requires reproducing internal density; density refinement ( 0→30→100 ) reproduces the target’s area density well, whereas occupancy-GD damages density and collapses the garment to an outline.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Additional examples of the main result ( 4 examples of test256 , point-cloud view). Columns: target (black), ours- 0 (gray), ours- 30 (blue), and random-r4- 30 (orange); C in each cell is the Chamfer distance. These correspond to the same 4 examples as Figure 3 .
Figure B.1: MNIST, rendered with Tu’s renderer ( 32×32 saturated occupancy) ( 10 examples). Columns are the same as Figure 6 (target / ours-0 / ours-30 / ours-100 / occupancy-GD-100 / Tu ( n=10 )).
Figure B.2: Fashion-MNIST, rendered with Tu’s renderer ( 32×32 saturated occupancy) ( 10 examples). Columns are the same as Figure 7 .
Figure B.3: MNIST, density maps (common grayscale), all balanced 50 examples. Columns are the same as Figure 6 .
Figure B.4: Fashion-MNIST, density maps (common grayscale), all balanced 50 examples. Columns are the same as Figure 7 .
Image restoration faces a fundamental tradeoff: methods that minimize error produce blurry reconstructions, while those that maximize perceptual quality yield sharp but less faithful images. Existing approaches either commit to a single operating point on this distortion perception (DP) frontier or require paired-data supervision, auxiliary models, or hyperparameter tuning of the sampler to access different points. We show that flow map models, a recent extension of flow matching for few-step sampling that learns an average field, implicitly define a one-parameter family of denoisers that continuously spans the DP frontier. The lookahead parameter t acts as a control knob between the MMSE and perceptual regimes. For Gaussian targets, we prove that varying t exactly recovers the optimal DP frontier; for natural images, we observe similar behavior empirically. Within a Plug-and-Play solver, the same mechanism extends to general inverse problems, where it controls a tradeoff between perceptual alignment and data consistency. Despite the lack of exact optimality guarantees in this setting, a single trained flow map spans the DP tradeoff, matching or exceeding specialized baselines at both extremes. Extensive experiments on CelebA (128×128) and AFHQ (256×256) across several linear and nonlinear inverse tasks validate our findings.
Nicolas Zilberstein, Morteza Mardani, Santiago Segarra
Nonlinear inverse problems often trade inexpensive but fragile first-order updates against curvature-aware methods such as Gauss-Newton and Levenberg-Marquardt, which obtain stronger directions by repeatedly solving Jacobian-based linearized systems. We propose a learned alternative: amortize local inverse geometry into a reusable reverse operator. Our framework learns a bidirectional surrogate, Deceptron, and deploys it through D-IPG (Deceptron Inverse-Preconditioned Gradient), an iterative solver that pulls residual-corrected measurement-space proposals back to latent space. The key mechanism is a Jacobian Composition Penalty (JCP), which trains the reverse Jacobian to act as a local left inverse of the forward Jacobian; its runtime counterpart, RJCP, measures the same inverse-consistency error along optimization trajectories. We prove that D-IPG is first-order equivalent to damped Gauss-Newton under local pseudoinverse consistency, with deviation controlled by composition error and conditioning. Across seven PDE inverse-problem benchmarks, D-IPG outperforms standard baselines, achieves 94.8% mean success across the six-problem reliability suite, and reaches comparable or better recovery quality at up to 77x lower inference-time solve cost on the main benchmarks.
Diffusion models have emerged as powerful generative priors for solving inverse imaging problems. However, their practical deployment is hindered by the substantial computational cost of slow, multi-step sampling. Although Consistency Models (CMs) address this limitation by enabling high-quality generation in only one or a few steps, their direct application to inverse problems has remained largely unexplored. This paper introduces a modified consistency sampling framework specifically designed for inverse problems. The proposed approach regulates the sampler's stochasticity through a measurement-consistency mechanism that leverages the degradation operator, thereby enforcing fidelity to the observed data while preserving the computational efficiency of consistency-based generation. Comprehensive experiments on the Fashion-MNIST and LSUN Bedroom datasets demonstrate consistent improvements across both perceptual and pixel-level metrics, including the Fréchet Inception Distance (FID), Kernel Inception Distance (KID), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM), compared with baseline consistency and diffusion-based sampling methods. The proposed method achieves competitive or superior reconstruction quality with only a small number of sampling steps.
Amirreza Tanevardi, Pooria Abbas Rad Moghadam, Seyed Mohammad Eshtehardian +2
Department of Electrical Engineering Sharif University of Technology Tehran, Iran · Electronics Research Institute Sharif University of Technology Tehran, Iran