Ultimate3D: Teaching Multimodal LLMs 3D Camera-Object Relations by Large-Scale Visual Instruction Data
Authors: Liu He, Yizhi Song, Ruiqi Xu, Baris Sarper Tezcan, Xiao Zeng, Albert Y. C. Chen, Lu Xia, Shashwat Verma, Sankalp Dayal, Min Sun, Cheng-Hao Kuo, Daniel Aliaga
Organizations: Amazon, USA. · Computer Science, Purdue University, 610 Purdue Mall, Lafayette, 47907, Indiana, USA.
Multimodal Large Language Models (MLLMs) struggle with accurately capturing camera-object relations, especially for object orientation, camera viewpoint, and camera shots. This stems from the fact that existing MLLMs are trained on images with limited diverse camera-object relations and corresponding textual descriptions. To address this, we propose a synthetic generation pipeline to create large-scale 3D visual instruction datasets. Our framework takes 3D assets as input and uses rendering and diffusion-based image generation models to create photorealistic images preserving precise camera-object relations. Additionally, large language models (LLMs) are used to generate text prompts for guiding visual instruction tuning and controlling image generation. We create Ultimate3D, a large-scale 3D-grounded 2D visual instruction dataset. Ultimate3D consists of 240K VQAs with precise camera-object annotations, and corresponding benchmark. MLLMs fine-tuned on our proposed dataset outperform commercial models by a large margin, achieving an average accuracy improvement of 29.7% on camera-object relation recognition tasks. Our code, dataset, and benchmark will contribute to broad MLLM applications.
Figures & tables
Figure 1: Generating Synthetic Visual Instruction Dataset . Our framework uses open-sourced 3D assets to generate photo-realistic images with precisely controlled camera-object relation. Corresponding text instructions are also generated by Large Language Model (LLM). The generated Ultimate3D dataset (240K) and benchmark (8K) advance baseline MLLM models (Gemma-4, Qwen3-VL, etc.) to outperform commercial MLLMs (GPT-5.5, Gemini-3.5, etc.) on camera-object relation recognition tasks. The left panel illustrates representative MLLM failures, while the right panel provides a direct same-image comparison between base and Ultimate3D-fine-tuned models.
Figure 2: Our Framework. Each shade box corresponds to a section in Sec. 3 . Given open-sourced 3D assets, our approach leverages a 3D Renderer to generate 3D visual priors ( Iβ ) preserving ground truth camera-object relation ( β ). Meanwhile, LLMs take 3D asset category to generate diverse image descriptions ( Timg ) as conditional guidance. Both the 3D visual priors and diverse text prompts are used to generate synthetic images ( Isyn ) by multiple ControlNet-based networks. Corresponding text QA instructions ( Tqa ) are generated by LLMs given the ground truth camera-object relation. Our Ultimate3D dataset and benchmark (i.e. pairs of Tqa and Isyn ) contribute to fine-tuning and evaluation of MLLMs.
Figure 3: 3D Visual Prior Rendering. With a general camera model, our method utilizes a 3D Renderer (e.g., Blender) to render multiple 3D visual priors given arbitrary camera-object relations. O : camera center; A : object center; D : camera-object distance.
Figure 4: Qualitative Results. We show responses by GPT-5.5, Gemini-3.5-Flash with high thinking, Gemma-4-31B-it, and finetuned Gemma-4-31B-it on the Ultimate3D benchmark. Each model is asked questions (in gray boxes) regarding object orientation, camera viewpoint, and camera-shot type, together with the input images on the left. The model responses illustrate that the Gemma-4-31B-it model finetuned on Ultimate3D can precisely recognize camera-object relations on synthetic images. GPT-5.5 and Gemini-3.5-Flash with high thinking sometimes provide correct answers, but also exhibit inconsistent predictions across relation types. The original Gemma-4-31B-it frequently fails to produce the correct camera-object relation labels, while the finetuned model provides substantially more accurate and task-relevant answers.
MLLMs
Orientation (%↑)
Viewpoint (%↑)
Camera-Shots (%↑)
Ultimate3D
MMVP
Ultimate3D
MMVP
Ultimate3D
MMVP
Both
Syn
Real
Both
Syn
Real
Syn
LLaVA-1.5-7B
18.3
11.1
22.5
25.0
31.8
38.4
25.3
29.2
46.0
66.7
LLaVA-1.6-7B
18.2
14.7
20.3
25.0
36.8
43.9
29.8
33.3
46.6
71.4
Table 1: Quantitative Comparisons. We evaluate open-source and closed-source MLLMs on the Ultimate3D and MMVP benchmarks. LLaVA and Llama models are fine-tuned using their original full fine-tuning recipes, while Gemma-4 and Qwen models are fine-tuned with LoRA-based parameter-efficient fine-tuning (for Qwen, the vision-language merger is also fully trained). Results report MLLM response accuracy ( % ) for object orientation, camera viewpoint, and camera-shot recognition. Superscripts indicate the fine-tuning setup: F full fine-tuning and L LoRA fine-tuning.
MLLMs
VQAv2 Goyal et al. (2017)
Ultimate3D *
MMVP *
LLaVA-1.6-13B Liu et al. (2024a)
80.00
29.70
37.03
Ulti3D (100%)
78.77
79.83
70.77
Ulti3D (50%) + Liu et al. (2024a) (50%)
80.01
74.72
65.80
GPT-4o
/
46.30
51.57
Claude-3.5-Sonnet
/
45.90
54.70
Table 2: Generalization. We finetune LLaVA-1.6-13B model using a mixture of both general VQA and camera-object relation VQA datasets. Without degradation on general VQA tasks, finetuned model improves its ability on camera-object relation recognition and beats SOTA (i.e. GPT-4o) by 28.42% accuracy on Ultimate3D benchmark. ∗ denotes the average accuracy ( % ) across all 3 types of camera-object relation VQAs in Tab. 1 .
Figure 5: Ablations on Image Generation Pipeline. Our framework shows superior performance by using multi-ControlNet with SDXL backbones, and introducing both Canny edges and depth maps as visual priors. It consistently delivers better robustness and quality compared to text-only SDXL and Ma et al. (2024a) . When these visual priors are removed (top-right: removing all visual priors; bottom-right: removing depth prior), the generator often fails to produce good results and brings lower generation success rate (see Sec 4.6 , Ma et al. (2024a) only achieves 55.11% success rate v.s. ours 93.07% )
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Limitations of Text-Image Aligning Metrics. We evaluate the ImageReward Xu et al. (2024) (IR) between the text promopt regarding object orientation with above 4 synthetic images. The image matches the correct camera-object relation text prompt (”front”) may reach the highest score. However, image with the highest matching score (it faces to ”left” direction) is not the correct one with expected orientation. This failure is prevailing across all categories.
Figure 7: Failures of Pose Estimation Model. Enlighten by Ma et al. (2024a) , we train a transformer-based pose estimation model on a 10-fold manner for generated images on each single category. The images with less prediction errors may indicate successful generation. However, the pose estimation model is not robust to complex objects and diverse camera-object relations in our dataset.
Ablations
CLIP-T ↑
CLIP-I ↑
DINO ↑
IR ↑
FID ↓
Ours (SD-V1.5)
29.80
83.56
69.36
-0.29
289.99
Ours (SDXL)
34.40
87.44
70.79
0.23
256.94
Appendix
Table 3: Ablations on DM Backbones . SDXL provides better fidelity (CLIP-I, DINO), realism (FID), and text-image alignment (CLIP-T, IR) than other alternatives.
Figure 8: Factors influencing Image Generation. The upper-left example shows that bottom-view camera-object relations can introduce physical anomalies, such as the tricycle appearing with its front lifted unnaturally. The upper-right example highlights that low-quality 3D assets can produce fragmented depth maps or Canny edge visual priors, resulting in unrealistic generated images. The lower-left example shows that small subjects viewed from a long-shot perspective may lead to duplicate subjects in the generated image. The lower-right example illustrates how complex 3D asset structures can cause texture blending issues, particularly where intricate geometry is present.
Figure 9: User Study UI Page. Users will see a pair of side-by-side images. The left hand side is an RGB image generated by Blender, and the right hand side is the synthetic image generated by DMs using that RGB image as guidance prior. A vivid preserving of 3D geometry and structure in the synthetic image is a success. Users are required to provide a binary answer regarding the success of the image generation.
Figure 10: Additional Qualitative Results with LLaVA. We evaluate camera-object relation recognition for GPT-4o, Claude-3-Sonnet, LLaVA-1.6-13B, and finetuned LLaVA-1.6-13B on the Ultimate3D benchmark. Each model is asked questions, shown in gray boxes, about object orientation, camera viewpoint, and camera-shot type, together with the input images on the left. The responses show that finetuning LLaVA-1.6-13B on Ultimate3D substantially improves camera-object relation recognition compared with the base LLaVA model and closed-source models.
Figure 11: Additional Qualitative Results with Gemma. We evaluate camera-object relation recognition for GPT-5.5, Gemini-3.5-Flash under the high thinking setting, Gemma-4-31B-it, and fine-tuned Gemma-4-31B-it on the Ultimate3D benchmark. Each model is asked questions, shown in gray boxes, about object orientation, camera viewpoint, and camera-shot type, together with the input images on the left. The responses show that fine-tuning Gemma-4-31B-it on Ultimate3D substantially improves camera-object relation recognition compared with the base Gemma-4-31B-it and closed-source models.
Model
Orientation
Viewpoint
Camera shot
Qwen3.5-2B
17.2 → 32.4
33.3 → 45.2
40.9 → 49.7
Qwen3.5-9B
20.7 → 34.3
41.3 → 46.9
42.4 → 48.8
Qwen3-VL-8B
20.5 → 34.4
37.8 → 50.1
43.6 → 51.8
Gemma-4-E2B-it
12.25 → 27.17
31.67 → 43.25
39.08 → 46.58
Gemma-4-E4B-it
16.83 → 28.42
40.00 → 46.33
41.08 → 49.58
Gemma-4-26B-A4B-it
16.83 → 34.08
43.42 → 47.33
42.33 → 51.42
Appendix
Table 4: Accuracy (%) on the answer-balanced ImageNet3D evaluation set, reported as base → Ultimate3D-fine-tuned. Chance is 12.5% for orientation and 33.3% for viewpoint and camera shot.
Relation
Class
Synthetic
PASCAL3D+
MEBOW
ImageNet3D
Orientation
Back
79–91
48–62
48–68
28–43
Back Left
76–89
43–61
45–70
26–43
Left
87–89
74–85
33–47
35–53
Front Left
77–90
54–73
29–39
24–35
Front
83–89
66–71
28–34
29–39
Front Right
72–85
62–69
30–39
20–48
Appendix
Table 5: Per-class accuracy (%) after Ultimate3D fine-tuning; each cell is the range over Qwen3.5-2B, Qwen3.5-9B and Qwen3-VL-8B. PASCAL3D+ carries no camera-shot annotation and MEBOW only orientation. Rear-oblique orientations, Bottom and Long-shot are the residual weak classes.
Real photographs
Synthetic
Condition
F
S
F
S
Subtle directional / elevation cue
52
27
41
28
Pose near a class boundary
47
28
41
27
Cluttered background
37
18
18
13
Small object ( <15% of frame)
24
15
36
20
Shape-symmetric object
9
2
6
3
Appendix
Table 6: Prevalence (%) of image conditions among images all three fine-tuned models answered incorrectly (F) and matched images all three answered correctly (S); n=90 and n=60 per column pair. Real = PASCAL3D+ and MEBOW.
Figure 12: Representative failures, part 1 of 2: geometric cues in the image plane. Every image was answered incorrectly by all three fine-tuned models; panels give the ground-truth label (green) and the answer of Qwen3.5-9B + Ultimate3D (red). Boundary: the true azimuth is within a few degrees of a bin edge. Symmetry: front and back are barely distinguishable and the prediction is inverted. Rear-oblique: a shallow rear three-quarter view is answered with the neighbouring side bin. Elevation, framing and people failures are shown in Fig. 13 .
Figure 13: Representative failures, part 2 of 2: viewpoint, framing and people; continued from Fig. 12 . Every image was answered incorrectly by all three fine-tuned models; panels give the ground-truth label (green) and the answer of Qwen3.5-9B + Ultimate3D (red). Elevation: the camera is above or below but the cue is faint. Framing: the object fills the frame uncropped, at the Close-up/Medium-shot threshold. People: clutter, small scale, or a head facing away from the torso.
Figure 14: Diversity of Ultimate3D . Our Ultimate3D dataset and benchmark cover 100 categories of objects, range diverse camera-object relation settings, and provide plausible image quality. (Each row shows images with the same orientation but in diverse subject and context.)