Generalization over Memorization: Generalization-Aware Diffusion Adaptation for Single-Image Multi-View Synthesis

Jie Li; Xingchen Zou; Yuxuan Liang

cs.CVAug 29, 2026

Generalization over Memorization: Generalization-Aware Diffusion Adaptation for Single-Image Multi-View Synthesis

Authors: Jie Li, Xingchen Zou, Yuxuan Liang

Organizations: Huazhong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

We present the winning solution to the ACM Multimedia 2026 Grand Challenge on Single-Image Guided Multi-Angle Image Synthesis. It ranks first among 293 registered teams; 56 teams obtained at least one scored submission on the public Phase-A leaderboard. With only 40 training scenes, the challenge requires 26 target views from one RGB model and one forward pass per view; it prohibits explicit geometry, external rendering, chained generation, candidate selection, and post-processing. We identify a critical model-selection failure: shared training and validation scenes make memorization appear as transferable view control. We therefore introduce GoM. Short for Generalization over Memorization, the framework combines scene-disjoint validation, exposure-matched selection, and targeted diffusion adaptation. Its synthesis model adapts a 4B rectified-flow DiT using rank-32 LoRA, optimizer restarts, late-checkpoint averaging, and VAE decoder tuning. More than 300 offline experiments and 24 online submissions show that validation design and training-trajectory control can matter as much as architecture scale in small-data generative modeling.

Figures & tables

Figure 1. Qualitative results. From a center-medium input, B2 synthesizes bottom-right-far and top-far views; predictions are paired with ground truth across three scenes. Three rows of qualitative results. Each row shows one input followed by ground-truth and model outputs for bottom-right-far and top-far target views. The scenes depict a basketball court, a pavilion, and a forest.

Figure 2. Overview of GoM: (A) transfer-aware model selection, (B) staged LoRA and decoder adaptation, and (C) single-model inference from one source to requested views without geometry or post-processing. A three-panel overview of Generalization over Memorization. Panel A shows generalization-aware model selection. Panel B shows staged LoRA training with checkpoint averaging and two-step decoder adaptation. Panel C uses real experimental images to show deterministic inference from one center-medium source image to six representative requested views.

Intervention axis	Offline $\Delta$	Online $\Delta$	$r_{a}$
Checkpoint averaging	$+0.0202$	$+0.0169$	84.0%
Decoder adaptation	$+0.0114$	$+0.0089$	78.1%
Seed selection	$+0.0170$	$+0.0092$	54.1%
Adam restarts	$+0.0483$	$+0.0109$	22.6%

Table 1. Offline-to-online transfer calibration from paired comparisons.

Setting	B1	B2
LoRA average	epochs 130/140/150	epochs 120/130/140/150
Decoder data	34-scene reconstruction	40 scenes $+$ 11,273 open images
Sampling	guidance 2.5, 14 steps	guidance 1.75, 10 steps
Time per view	12.2 s	$\sim$ 9.0 s
Training time	44.5 h	45.0 h
Peak memory	21.4 GB	21.4 GB

Table 2. Final submission configurations and measured resource profile.

Entry	Total $\uparrow$	P/40 $\uparrow$	SSIM $\uparrow$	$1{-}$ LPIPS $\uparrow$	NRIQA $\uparrow$	Rank
GoM B2 (Ours)	1.8628	0.3051	0.3282	0.4509	0.7786	1st [-1pt] Winner
Rank-2 team	1.8542	0.3013	0.3254	0.4698	0.7577	2nd
Rank-3 team	1.8488	0.3090	0.3197	0.4654	0.7547	3rd
B1 (Phase B)	1.8512	0.2998	0.3211	0.4557	0.7746	—
B1 (Phase A)	1.9386	0.3001	0.3749	0.4805	0.7831	—

Table 3. Official Phase-B top three and GoM diagnostics. Bold marks the best component among ranked teams; diagnostic rows do not receive separate ranks.

Intervention	Paired online scores	$\Delta$
Seed selection	$1.8848\to 1.8940$	$+0.0092$
Checkpoint averaging	$1.8940\to 1.9109$	$+0.0169$
Adam restarts	$1.9136\to 1.9245$	$+0.0109$
Decoder adaptation (restart)	$1.9245\to 1.9334$	$+0.0089$
Decoder adaptation (continuous)	$1.9136\to 1.9225$	$+0.0089$
Three-point late averaging	$1.9245\to 1.9314$	$+0.0069$

Table 4. Paired Phase-A evidence; rows are independently controlled and non-additive.

Family	Representative controlled evidence
Capacity	A 9B backbone reaches 1.8451 and rank-128 LoRA reaches 1.8055; neither exceeds the matched 4B rank-32 baseline.
Data/prompt	Flips change the score by $-0.103$ , all-pairs curricula by about $-0.11$ , and prompt suffixes by $-0.056$ .
Objectives	High-weight evaluator-directed NRIQA losses reduce SSIM to 0.24–0.29; SGDR changes the score by $-0.14$ and DPO by $-0.046$ ( Loshchilov and Hutter, 2017 ; Rafailov et al., 2023 ) .
Composition	Expert routing is $-0.0042$ online; TIES/DARE merging scores 1.6047/1.5856 ( Yadav et al., 2023 ; Yu et al., 2024 ) ; encoder tuning lowers end-to-end performance by 0.019.

Table 5. Representative falsifications; values are six-scene audit scores or deltas unless marked online.

Multi-View Image Synthesis

See all 24 papers

NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis
Oct 3, 2026Ramil Khafizov, Ilya Statsenko, Ruslan Rakhimov +3Multi-View Image Synthesis Novel View Synthesis
Applied AI Institute · T-Tech · KAUST +1
Novel View Synthesis using DDIM Inversion
Aug 14, 2025Sehajdeep Singh, A V Subramanyam, Aditya Gupta +1Novel View Synthesis Diffusion Model Guidance
Indraprastha Institute of Information Technology, Delhi
VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis
Sep 27, 2026Kangjie Chen, Xiangyu Li, Dongbin Zhang +12Novel View Synthesis Conditional Diffusion Models
XPeng Motors · The Chinese University of Hong Kong · Tsinghua University

Oct 3, 2026J/K move · Enter open · S save

NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis

Ramil Khafizov, Ilya Statsenko, Ruslan Rakhimov, Artem Komarichev+2

Applied AI Institute · T-Tech · KAUST+1

Multi-View Image Synthesis Novel View Synthesis Autoregressive Image Generation Multi-View Geometry

Open paper AskPDF arXiv

Sparse-view novel view synthesis is a central problem in 3D content creation, but diffusion-based approaches remain limited by iterative denoising, making multi-view generation expensive at inference time. We introduce NAMVIS, a diffusion-free framework that reformulates multi-view image synthesis as geometry-conditioned next-scale autoregression. Instead of generating target views through repeated denoising, NAMVIS predicts discrete visual tokens through a small number of coarse-to-fine scale steps, while sampling all tokens within each scale and across target views in parallel. To anchor this generation process to explicit camera geometry, we propose Multi-scale Projective Pose Encoding, which injects source and target camera transformations into both target-view self-attention and source-to-target cross-attention at every resolution. NAMVIS further combines global conditioning with dense geometry-aware cross-attention, enabling the model to preserve source-view appearance while maintaining target-view consistency. Across Objaverse, GSO, and OmniObject3D, NAMVIS outperforms diffusion-based baselines in PSNR, SSIM, and LPIPS, while running over 3 times faster than the evaluated diffusion baselines under the same evaluation setting. These results suggest that geometry-conditioned next-scale autoregression is a promising and efficient alternative to diffusion for sparse-view multi-view synthesis. Additional qualitative results, videos, and resources are available at https://corl-team.github.io/namvis/

Abstract

Multi-View Image Synthesis

Explore similar work

NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis

Novel View Synthesis using DDIM Inversion

VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis