Perception-Aware Multimodal Spatial Reasoning from Monocular Images
Organizations: NUS, Singapore · A*STAR, Singapore · NTU, Singapore · MIT, US
Abstract
Spatial reasoning from monocular images is essential for autonomous driving, yet current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception, particularly under large scale variation and ambiguous object appearance. We propose a simple yet effective perception-aware multimodal reasoning framework that equips VLMs with explicit object-centric grounding ability. Instead of relying on textual bounding-box outputs, each referred object is represented using all Visual Reference Tokens (VRTs) within its spatial extent, enabling visual evidence and textual reasoning to be processed jointly in a unified token space. To further strengthen cross-modal interaction, we construct a Multimodal Chain-of-Thought (MM-CoT) dataset that injects aligned visual and textual reasoning signals. A deterministic ordering strategy is introduced to make supervision over inherently unordered VRT sets fully compatible with the VLM's autoregressive next-token prediction. With only standard supervised fine-tuning, our method achieves substantial improvements on the SURDS benchmark, outperforming previous approaches - including those using RL-based post-training - by a large margin across both single-object and multi-object tasks. These results demonstrate that accurate perception and multimodal reasoning are mutually reinforcing, and together form the key to robust spatial understanding in challenging monocular driving scenarios.
Figures & tables
| Model | Single-object | Multi-object | Score | ||||
| Yaw | Pixel | Depth | Dis | L/R | F/B | ||
| Random | 5.73 | 1.12 | 34.27 | 8.76 | 11.57 | 11.89 | 12.22 |
| GPT-4o | 13.08 | 1.62 | 2.49 | 11.57 | 47.89 | 3.14 | 13.30 |
| GPT-4o-mini | 3.24 | 0.28 | 0.22 | 4.22 | 21.51 | 2.05 | 5.25 |
| Gemini-1.5-pro | 19.14 | 4.41 | 22.70 | 61.95 | 66.38 | 22.05 | 32.77 |
| Gemini-2.0-flash | 9.30 | 5.41 | 32.97 | 69.30 | 77.30 | 20.00 | 35.71 |
| Row | Init W | CoT | Single-object | Multi-object | Score | ||||
| Yaw | Pixel | Depth | Dis | L/R | F/B | ||||
| 1 | QWen | T | 6.27 | 3.81 | 27.68 | 17.84 | 14.81 | 10.49 | 13.48 |
| 2 | QWen | MM | 23.54 | 1.27 | 24.88 | 54.69 | 68.24 | 29.42 | 33.67 |
| 3 | PaDT | T | 27.95 | 18.31 | 93.19 | 53.19 | 79.14 | 54.32 | 54.35 |
| 4 | PaDT | MM | 49.11 | 19.23 | 95.39 | 77.59 | 87.46 | 79.64 | 68.07 |
Visual Spatial Reasoning
Visual spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and physical change from images and video.
See all 251 papers