cs.CVNov 18, 2026

Ultimate3D: Teaching Multimodal LLMs 3D Camera-Object Relations by Large-Scale Visual Instruction Data

Authors: Liu He, Yizhi Song, Ruiqi Xu, Baris Sarper Tezcan, Xiao Zeng, Albert Y. C. Chen, Lu Xia, Shashwat Verma, 

Organizations: Amazon, USA. · Computer Science, Purdue University, 610 Purdue Mall, Lafayette, 47907, Indiana, USA.

Abstract

Multimodal Large Language Models (MLLMs) struggle with accurately capturing camera-object relations, especially for object orientation, camera viewpoint, and camera shots. This stems from the fact that existing MLLMs are trained on images with limited diverse camera-object relations and corresponding textual descriptions. To address this, we propose a synthetic generation pipeline to create large-scale 3D visual instruction datasets. Our framework takes 3D assets as input and uses rendering and diffusion-based image generation models to create photorealistic images preserving precise camera-object relations. Additionally, large language models (LLMs) are used to generate text prompts for guiding visual instruction tuning and controlling image generation. We create Ultimate3D, a large-scale 3D-grounded 2D visual instruction dataset. Ultimate3D consists of 240K VQAs with precise camera-object annotations, and corresponding benchmark. MLLMs fine-tuned on our proposed dataset outperform commercial models by a large margin, achieving an average accuracy improvement of 29.7% on camera-object relation recognition tasks. Our code, dataset, and benchmark will contribute to broad MLLM applications.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Beyond Single Object: Learning 3D Relations with Large Language Models

    Aug 16, 2026Kohsuke Ide, Ryousuke Yamada, Yue Qiu +43D VLMs3D Spatial Reasoning

  2. Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

    Sep 29, 2026Jaewoo Jung, Hyeonseo Yu, Honggyu An +103D Spatial Reasoning3D Scene Representation

  3. Lens3D: Target-Conditioned Visual Foveation for Fine-Grained 3D Understanding

    Oct 5, 2026Junming Huang, Zini Chen, Shuaiying Hou +3Active View Selection3D Visual Question Answering