cs.CVSep 29, 2026
SaveImagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
Organizations: KAIST AI · ETH Zürich · Google · TUM · ETH AI Center
Abstract
Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.
Figures & tables
Figure 1: Overall architecture of Imagine3D-LLM. Given multi-view images of a scene and a text instruction, our model appends a compact set of learnable Gaussian summary tokens between the image and text tokens. At an intermediate layer , the hidden states at the Gaussian summary token positions are decoded into a compact 3D Gaussian Splatting representation that abstractly reconstructs the scene. The model is trained jointly with a standard language modeling loss on the text output, a photometric reconstruction loss obtained by rendering the predicted Gaussians at the input viewpoints, and a distillation loss from a pretrained compact Gaussian teacher [ 1 ] .
Figure 2: Effects of joint reconstruction training on the LLM’s image features. (a) Image-to-image attention maps between a query point ( red dot) and other input views. Our model exhibits substantially stronger cross-view correspondences than the baseline, indicating that the Gaussian summary tokens implicitly drive the image features to align across views. (b) PCA visualization of the LLM’s image features (first three components shown as RGB). Our features are noticeably cleaner and more semantically structured, with corresponding objects (e.g., chairs, desks) encoded with consistent colors across views, reflecting the emergence of object-level abstractions.
Figure 3: Visualization of clustered Gaussian from summary tokens. We visualize the Gaussians clustered from a set of Gaussian summary tokens via K-means over the summary tokens ( cyan indicates the Gaussians belonging to one cluster). Similar summary tokens are mapped to the same object without any explicit clustering supervision.
| Method | SQA3D | Real-3DQA | ScanQA | ||||||
| EM | EM-R | EM | EM-R | CIDEr | BLEU-4 | METEOR | ROUGE | EM | |
| Specialists | |||||||||
| SQA3D [ 52 ] | 46.6 | – | – | – | – | – | – | – | – |
| ScanQA [ 3 ] | – | – | – | – | 64.9 | 10.1 | 13.1 | 33.3 | 21.1 |
| 2D MLLMs | |||||||||
| InternVL2-8B [ 67 ] | 33.0 | 45.3 | – | – | 62.5 | 3.3 | 14.5 | 34.3 | – |
Table 1: Quantitative results on 3D question-answering over SQA3D [ 52 ] , Real-3DQA [ 51 ] , and ScanQA [ 3 ] . “Specialists” refer to models tailored to individual tasks via task-specific decoders. General-purpose 2D MLLMs [ 67 , 75 , 89 ] are reported under a zero-shot protocol. We report values available from prior works; “–” marks entries unavailable to us.
| Method | Scan2Cap (IoU@0.5) | ScanRefer | Multi3DRefer | |||||
| ROUGE | BLEU-4 | METEOR | CIDEr | Acc@0.25 | Acc@0.5 | F1@0.25 | F1@0.5 | |
| Specialists | ||||||||
| Scan2Cap [ 11 ] | 44.5 | 23.3 | 22.0 | 35.2 | – | – | – | – |
| 3DJCG [ 8 ] | 50.8 | 31.0 | 24.2 | 49.5 | 49.6 | 37.3 | – | 26.6 |
| ScanRefer [ 10 ] | – | – | – | – | 37.3 | 24.3 | – | – |
| M3DRef-CLIP [ 88 ] | – | – | – | – | 51.9 | 44.7 | 42.8 | 38.4 |
Table 2: Evaluation of 3D dense captioning and visual grounding on Scan2Cap [ 11 ] , ScanRefer [ 10 ] , and Multi3DRefer [ 88 ] .
| Model | SPAR-Bench | |||
| Avg. | Low | Med. | High | |
| Proprietary | ||||
| GPT-4o [ 34 ] | 36.4 | 29.3 | 24.9 | 45.1 |
| Claude-3.7-Sonnet [ 2 ] | 21.8 | 25.4 | 7.3 | 23.3 |
| General-purpose 2D MLLMs | ||||
| LLaVA-Video-7B [ 89 ] | 32.3 | 23.6 | 24.8 | 42.6 |
Table 3: Evaluation on SPAR-Bench [ 87 ] .
| Method | SQA3D | ScanQA | Scan2Cap |
| Baseline | 56.5 | 26.2 | 63.1 |
| Imagine3D-LLM (Ours) | 63.8 | 29.9 | 67.6 |
| Method | ScanRefer | Multi3DRefer | SPAR |
| Baseline | 58.3 | 57.4 | 60.9 |
| Imagine3D-LLM (Ours) | 62.8 | 60.2 | 68.5 |
Table 4: Controlled comparison against the base MLLM. Both models share the identical backbone, training data, and schedule; the baseline is trained with standard visual instruction tuning, without the Gaussian summary tokens or the reconstruction and distillation objectives.
| Layer | SPAR-Bench | |
| 7 (early) | 60.7 | 60.2 |
| 14 (middle, Ours) | 63.8 | 68.5 |
| 21 (late) | 61.7 | 64.0 |
Table 5: Effect of the layer from which Gaussian tokens are decoded.
| Method | SPAR-Bench | |
| Baseline (1 ep.) | 56.5 | 60.9 |
| Baseline (2 ep.) | 57.2 | 61.5 |
| Baseline (4 ep.) | 52.3 | 55.2 |
| Recon. only (1 ep.) | 57.7 | 57.6 |
| Recon. only (2 ep.) | 61.9 | 65.1 |
| Recon. only (4 ep.) | 63.7 | 67.9 |
Table 6: Effect of distillation from the compact Gaussian teacher.
| Variant | SPAR-Bench | |
| Baseline | 56.5 | 60.9 |
| + Teacher tokens | 56.4 | 60.8 |
| + Distill. only | 56.6 | 60.5 |
| + Recon. (4 ep.) | 63.7 | 67.9 |
| Full (Ours, 1 ep.) | 63.8 | 68.5 |
Table 7: Isolating the source of the gains. + Teacher tokens : the teacher’s query tokens are fed directly to the LLM as input. + Distill. only : summary tokens supervised by but without .
| SPAR-Bench | ||
| 1296 | 61.9 | 64.6 |
| 2592 (Ours) | 63.8 | 68.5 |
| 5184 | 60.3 | 58.4 |
Table 8: Number of Gaussian summary tokens .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Layer-wise cross-view correspondence during training. We report cross-view correspondence scores for image-token representations extracted from middle LLM layers on SQA3D [ 52 ] test samples. Scores increase consistently throughout training, with pronounced gains near the Gaussian decoding layer ( ), indicating that joint reconstruction training improves the view-consistency of intermediate image features. Shaded regions indicate variance across SQA3D [ 52 ] test samples.
| Method | mIoU |
| Baseline (no summary tokens) | 26.54 |
| Imagine3D-LLM (Ours) | 35.43 |
| DINOv3 [ 65 ] | 37.62 |
Table 9: Unsupervised semantic segmentation probe. We cluster the LLM’s image features via K-means and measure mIoU against ground-truth ScanNet labels. DINOv3 [ 65 ] is evaluated under the same protocol as an upper-bound reference.
| Method | PSNR ( ) | SSIM ( ) | LPIPS ( ) |
| Gaussian teacher [ 72 ] | 20.60 | 0.76 | 0.44 |
| Imagine3D-LLM (Ours) | 17.61 | 0.74 | 0.49 |
Table 10: Reconstruction quality on SQA3D test scenes. We compare the rendering quality of our Gaussian summary tokens against the compact Gaussian teacher [ 72 ] used during training. Higher is better for PSNR and SSIM; lower is better for LPIPS.
Figure 5: Visualization of attention maps between multi-view images and PCA visualizations of the image features. (a) Image-to-image attention maps between a query point ( red dot) and other input views. Our model exhibits substantially stronger cross-view correspondences than the baseline, indicating that the Gaussian summary tokens implicitly drive the image features to align across views. (b) PCA visualization of the LLM’s image features (first three components shown as RGB). Our features are noticeably cleaner and more semantically structured, with corresponding objects (e.g., chairs, desks) encoded with consistent colors across views, reflecting the emergence of object-level abstractions.
Figure 6: Visualization of clustered Gaussian from summary tokens. We visualize the Gaussians clustered from a set of Gaussian summary tokens via K-means over the summary tokens. Similar summary tokens are mapped to the same object without any explicit clustering supervision.
Figure 7: Visualization of attention maps between Gaussian summary tokens and images. We visualize the attention score map between a Gaussian summary token (a red dot indicating the decoded Gaussian) and the input images. Each Gaussian token attends to corresponding regions of the same object across multiple views.
| Training setting | Peak VRAM (GB) |
| w/o | 40 |
| Imagine3D-LLM (full) | 44 |
Table 11: Training-time computation cost. Peak per-GPU VRAM measured during training under identical batch size and sequence length. The reconstruction and distillation losses introduce only a modest memory overhead while substantially improving 3D reasoning.
| Method | Peak Memory (GB) | Inference Time (ms) |
| Baseline (7B) | 17.09 | 112.1 |
| VLM3R-7B [ 23 ] | 25.29 | 344 |
| Imagine3D-LLM (7B) | 20.43 | 176.9 |
Table 12: Inference cost comparison. Peak GPU memory and per-sample inference time measured on the SPAR evaluation set. VLM3R-7B [ 23 ] additionally runs an external 3D foundation model (CUT3R [ 76 ] ) on all input frames before fusion.
Figure 8: Visualization of reconstructed scene with estimated Gaussians. (Top) : input multi-view images. (Bottom) : a view rendered from the 3D Gaussians decoded by our Gaussian summary tokens. Although the rendering is coarse due to the compact token bottleneck, it preserves the overall scene layout and the placement of major objects, indicating that the tokens encode a coherent 3D abstraction of the scene.