Can Learning to Answer Spatial Questions Improve Novel-View Generation?

Jahyeok Koo; Hyeonseo Yu; Seungho Jang; Jaewoo Jung; Jiwon Kang; Seokju Cho; Seungryong Kim

cs.CVOct 14, 2026

Can Learning to Answer Spatial Questions Improve Novel-View Generation?

Authors: Jahyeok Koo, Hyeonseo Yu, Seungho Jang, Jaewoo Jung, Jiwon Kang, Seokju Cho, Seungryong Kim

Organizations: KAIST AI

Abstract

Generating novel views from limited observations requires preserving both the appearance and spatial structure of a scene. Although recent approaches exploit fine-grained geometric cues, they do not explicitly model high-level semantic and spatial properties such as object identity, attributes, and inter-object relationships. In this work, we investigate whether 3D language understanding can provide complementary supervision for novel-view generation. We introduce Ask4NV, built on a pretrained unified multimodal model with a Mixture-of-Transformers (MoT) architecture. Our model incorporates camera geometry into both VAE and ViT representations and leverages understanding features during generation through joint self-attention with a modality-aware attention masking strategy. Experiments demonstrate strong image quality and geometric consistency in novel-view generation, alongside competitive 3D language understanding. Further analysis shows that joint training improves cross-view correspondence in VAE representations and similarity in a 3D-aware feature space.

Figures & tables

Figure 1: We present Ask4NV , which enhances novel-view generation through 3D language understanding. Within a Mixture-of-Transformers architecture, 3D language understanding supervision trains the understanding transformer to interpret scene semantics and spatial relationships. During novel-view generation, joint self-attention combines this knowledge with the generation transformer’s visual synthesis capabilities. Ask4NV achieves higher PSNR than the baseline trained solely on novel-view generation on both in-domain and out-of-domain datasets.

Figure 2: Shared scene properties across 3D language understanding and novel-view generation. Questions about object categories, attributes, and spatial relationships address the same scene properties that should be represented in novel views. The knowledge required to answer these questions can provide complementary semantic and spatial supervision for novel-view generation.

Figure 3: Overall architecture of Ask4NV. Given source images and camera parameters of the source and target views, (a) for novel-view generation, the images are first encoded into VAE and ViT features by their respective visual encoders, and camera geometry is incorporated into these representations before they are passed to the transformers. (b) The generation transformer processes the camera-aware VAE tokens, while the understanding transformer processes the camera-aware ViT tokens together with text tokens for 3D language understanding. (c) During novel-view generation, these modality-specific tokens interact through joint self-attention under a unified modality-aware attention mask.

Figure 4: Layer-wise cross-view correspondence. We report PCK@1, PCK@2, and PCK@4 across Transformer layers on ScanNet ( Dai et al., 2017 ) for Lance-FT ( Fu et al., 2026 ) and Ask4NV. Layer $l=26$ exhibits the strongest correspondence across all PCK thresholds.

Dataset	Method	PCK@1	PCK@2	PCK@4
ScanNet	Lance-FT	30.30	42.03	58.22
	Lance-FT + ViT	38.77	54.93	72.09
	Ask4NV (Ours)	46.13	60.73	74.99
ScanNet++	Lance-FT	33.34	44.31	60.28
	Lance-FT + ViT	31.40	44.91	63.87
	Ask4NV (Ours)	43.27	55.25	69.55

Table 1: Cross-view Correspondence at Layer 26. We report PCK@1, PCK@2, and PCK@4 on ScanNet and ScanNet++ for Lance-FT, Lance-FT + ViT and Ask4NV.

Method	DL3DV	ScanNet	Re10K	ScanNet++
Lance-FT	0.512	0.498	0.580	0.514
Lance-FT + ViT	0.546	0.555	0.636	0.538
Ask4NV (Ours)	0.599	0.557	0.680	0.557

Table 2: Patch-wise feature similarity between generated and ground-truth views. We report feature similarity on DL3DV, ScanNet, RealEstate10K (Re10K), and ScanNet++.

Dataset	Method	2D Metrics			3D Metrics
Dataset	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ATE $\downarrow$	RPE r $\downarrow$	RPE t $\downarrow$	MEt3R $\downarrow$
DL3DV	CAT3D † ( Gao et al., 2024 )	12.662	0.287	0.489	0.777	15.59	1.674	0.446
	Matrix3D ( Lu et al., 2025 )	13.997	0.344	0.412	0.351	4.80	0.691	0.358
	NVComposer ( Li et al., 2025 )	11.532	0.255	0.560	0.995	32.34	1.748	0.511
	CAMEO ( Kwon et al., 2025 )	12.991	0.312	0.463	0.628	13.16	1.174	0.388
	Omni-View ( Hu et al., 2026 )	13.009	0.221	0.558	1.226	27.86	2.514	0.511

Table 3: Evaluation of novel view generation. We evaluate on DL3DV and ScanNet (in-domain for Ask4NV) and ScanNet++ and Re10K (out-of-domain for Ask4NV). We report metrics for both 2D image fidelity and 3D geometric consistency.

\dagger

denotes the reproduced CAT3D model. Bold and underline indicate the best and second-best, respectively.

Figure 5: Qualitative results of novel view generation of Ask4NV and baselines on in-domain DL3DV ( Ling et al., 2024 ) and out-of-domain ScanNet++ ( Yeshwanth et al., 2023 ) . Blue and red cameras indicate source and target viewpoints, respectively.

2D LMMs
Method	Size	SQA3D ${}_{\text{test}}$		ScanQA ${}_{\text{val}}$					ScanRefer ${}_{\text{val}}$		Multi3DRefer ${}_{\text{val}}$
Method	Size	EM	EM-R	C	B-4	M	R	EM	[email protected]	[email protected]	[email protected]	[email protected]
InternVL2-8B ( Chen et al., 2024b )	8B	33.0	45.3	62.5	3.3	14.5	34.3	–	–	–	–	–
Qwen2.5-VL-3B ( Bai et al., 2025 )	3B	46.3	49.0	67.7	9.7	12.9	35.6	23.9	–	–	–	–
LLaVA-Video-7B ( Zhang et al., 2024 )	7B	48.5	–	88.7	3.1	17.7	44.6	–	–	–	–	–
Lance-3B ( Fu et al., 2026 )	3B	23.4	37.3	6.0	0.3	11.4	9.9	0.3	–	–	–	–

Table 4: Evaluation of 3D language understanding. We provide quantitative comparison of Ask4NV and baselines on 3D question answering (SQA3D ( Ma et al., 2022 ) and ScanQA ( Azuma et al., 2022 ) ) and 3D visual grounding (ScanRefer ( Chen et al., 2020 ) and Multi3DRefer ( Zhang et al., 2023 ) ). Results are reported from prior works, and “-” indicates an unavailable result. C, B-4, M, and R denote CIDEr, BLEU-4, METEOR, and ROUGE, respectively.

Views	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Source	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
$N=2$	Lance-FT	13.001	0.294	0.459
$N=2$	Ask4NV (Ours)	13.900	0.329	0.388
$N=3$	Lance-FT	13.292	0.298	0.430
$N=3$	Ask4NV (Ours)	14.559	0.344	0.355

Table 5: Ablation on different numbers of source views.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Unidirectional	13.653	0.314	0.424
Bidirectional (Ours)	14.939	0.355	0.313

Table 6: Ablation on bidirectional VAE attention.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Joint	Gen-ViT	Und-VAE	2D Metrics			3D Metrics
Dataset	Joint	Gen-ViT	Und-VAE	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ATE $\downarrow$	RPE r $\downarrow$	RPE t $\downarrow$	MEt3R $\downarrow$
DL3DV	✗	✗	✗	13.291	0.297	0.432	0.185	3.80	0.354	0.361
	✗	✓	✗	13.878	0.321	0.384	0.267	5.37	0.498	0.367
	✓	✓	✓	13.482	0.313	0.421	0.380	5.96	0.681	0.407
	✓	✓	✗	14.939	0.355	0.313	0.158	3.11	0.298	0.349
ScanNet	✗	✗	✗	14.292	0.530	0.518	0.233	15.45	0.351	0.398

Appendix

Table 7: Ablation on joint training settings for novel-view generation. All Gen-ViT configurations use the same generation-side camera conditioning.

Joint	Gen-ViT	Und-VAE	SQA3D ${}_{\text{test}}$		ScanQA ${}_{\text{val}}$					ScanRefer ${}_{\text{val}}$		Multi3DRefer ${}_{\text{val}}$
Joint	Gen-ViT	Und-VAE	EM	EM-R	C	B-4	M	R	EM	[email protected]	[email protected]	[email protected]	[email protected]
✗	✗	✗	54.3	57.3	86.5	13.6	17.2	42.9	24.8	52.6	47.0	53.1	48.5
✗	✗	✓	54.3	57.0	85.1	13.6	16.9	42.3	25.0	51.9	46.3	52.8	48.3
✓	✓	✓	55.4	58.4	84.3	11.8	16.6	42.6	25.6	53.4	47.4	53.2	48.6
✓	✓	✗	54.9	57.7	87.7	13.6	17.4	43.7	25.6	52.9	47.3	52.6	48.3

Appendix

Table 8: Ablation on joint training settings for 3D language understanding.

Figure 6: Cross-view correspondence accuracy throughout training.

Figure 7: Visualization of cross-view VAE feature similarity.

Dataset	ViT–VAE	Noise–Clean	2D Metrics			3D Metrics
Dataset	bidir.	bidir.	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	ATE $\downarrow$	RPE ${}_{\mathrm{{}_{r}}}$ $\downarrow$	RPE ${}_{\mathrm{{}_{t}}}$ $\downarrow$	MEt3R $\downarrow$
DL3DV	✗	✗	13.573	0.310	0.410	0.378	6.54	0.687	0.373
	✗	✓	13.945	0.326	0.384	0.319	6.28	0.607	0.363
	✓	✗	13.653	0.314	0.424	0.409	5.59	0.748	0.406
	✓	✓	14.939	0.355	0.313	0.158	3.11	0.298	0.349
ScanNet	✗	✗	15.542	0.557	0.446	0.200	13.03	0.348	0.420

Appendix

Table 9: Ablation on bidirectional attention between visual tokens.

Dataset	QA	Grounding	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
DL3DV	✓	✗	13.66	0.313	0.402
	✗	✓	13.03	0.291	0.453
	✓	✓	12.92	0.285	0.451
ScanNet	✓	✗	13.94	0.525	0.524
	✗	✓	13.50	0.514	0.552
	✓	✓	13.54	0.510	0.546

Appendix

Table 10: Ablation on the composition of 3D language understanding supervision.

Figure 8: Additional qualitative results of novel view generation on DL3DV, comparing Ask4NV with baseline methods. Blue and red cameras indicate source and target viewpoints, respectively.

Figure 9: Additional qualitative results of novel view generation on ScanNet, comparing Ask4NV with baseline methods. Blue and red cameras indicate source and target viewpoints, respectively.

Figure 10: Additional qualitative results of novel view generation on Re10K, comparing Ask4NV with baseline methods. Blue and red cameras indicate source and target viewpoints, respectively.

Figure 11: Additional qualitative results of novel view generation on ScanNet++, comparing Ask4NV with baseline methods. Blue and red cameras indicate source and target viewpoints, respectively.

Figure 12: Additional qualitative results of novel view generation on $N=3$ source views, comparing Ask4NV with baseline methods. Blue and red cameras indicate source and target viewpoints, respectively.

Figure 13: Additional qualitative results of novel view generation on $N=2$ source views, comparing Ask4NV with baseline methods. Blue and red cameras indicate source and target viewpoints, respectively.

Explore similar work

Thinking with Novel Views: A Systematic Analysis of Generative-Augmented Spatial Intelligence
May 11, 2026Yanbing Zhang, Bo Wang, Jianhui Liu +9Visual Spatial Reasoning VLM Reasoning
Joy Future Academy
Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs
Oct 4, 2026Yuqun Wu, Yao Xiao, Chuhang Zou +2Visual Spatial Reasoning 3D VQA
University of Illinois at Urbana-Champaign · Meta
GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space
Sep 28, 2026Kerui Ren, Tao Lu, Linning Xu +5Multi-View Consistency Novel View Synthesis
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +3

May 11, 2026J/K move · Enter open · S save

Thinking with Novel Views: A Systematic Analysis of Generative-Augmented Spatial Intelligence

Yanbing Zhang, Bo Wang, Jianhui Liu, Nan Jiang+8

Joy Future Academy

Visual Spatial Reasoning VLM Reasoning

Open paper AskPDF arXiv

Current Large Multimodal Models (LMMs) struggle with spatial reasoning tasks requiring viewpoint-dependent understanding, largely because they are confined to a single, static observation. We propose Thinking with Novel Views (TwNV), a paradigm that integrates generative novel-view synthesis into the reasoning loop: a Reasoner LMM identifies spatial ambiguity, instructs a Painter to synthesize an alternative viewpoint, and re-examines the scene with the additional evidence. Through systematic experiments we address three research questions. (1) Instruction format: numerical camera-pose specifications yield more reliable view control than free-form language. (2) Generation fidelity: synthesized view quality is tightly coupled with downstream spatial accuracy. (3) Inference-time visual scaling: iterative multi-turn view refinement further improves performance, echoing recent scaling trends in language reasoning. Across four spatial subtask categories and four LMM architectures (both closed- and open-source), TwNV consistently improves accuracy by +1.3 to +3.9 pp, with the largest gains on viewpoint-sensitive subtasks. These results establish novel-view generation as a practical lever for advancing spatial intelligence of LMMs.

Abstract

Explore similar work

Thinking with Novel Views: A Systematic Analysis of Generative-Augmented Spatial Intelligence

Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs

GeoVerse: World-Consistent Novel View Synthesis in Geometric Latent Space