cs.CVNov 17, 2026

Semantic-Guided Geometric Decoding for Token-Efficient Visual Grounding

Authors: Zhaojie Chen, Junyan Yang, Le Yang, Qi Dang, Yucheng Huang, Qianyi Che, Zhaoqiang Chen, Xinqi Zhu, 

Organizations: Huawei Technologies Co., Ltd.

Abstract

Multimodal large language models (MLLMs) provide a unified interface for language-conditioned visual perception. In pixel-to-sequence grounding, autoregressive coordinate generation requires multiple sequential decoding steps per box, increasing inference cost with the number of instances. Pixel-to-embedding methods reduce this overhead but lack explicit spatial encoding when using a shared special token. We introduce SemGeo, a token-efficient visual grounding framework that represents each instance with a single grid token for coarse localization. A geometric decoder reuses visual features and hidden states to recover precise discrete box coordinates in parallel. To train proposal generation and box localization together, we introduce Gaussian soft labels that encode coordinate distances during supervised fine-tuning (SFT). Reinforcement post-training further combines box-set rewards with coordinate supervision on model-generated proposals. This stage trains the language model and geometric decoder on their own predictions to mitigate the mismatch between teacher-forced training and inference. Across eight benchmarks spanning object detection, referring object detection, and document layout grounding, SemGeo achieves the highest F1@mIou among compared MLLMs on six benchmarks. It also delivers 2.07×2.07\times and 1.25×1.25\times the mean box-generation throughput of representative grounding baselines under vLLM and Transformers, respectively, while achieving higher accuracy in both comparisons.

Figures & tables

Explore similar work

CardsList
  1. IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

    Jul 17, 2026Xiuyuan Zhu, Ke Lu, Hao Wu +4Cross-Modal Knowledge DistillationMultimodal Large Language Models

  2. Jag: Direct Box Prediction for Efficient Visual Grounding

    Oct 9, 2026Xiuyuan Zhu, Ke Lu, Hao Wu +3Visual GroundingVision-Language Grounding

  3. Hi-Token: Hierarchical Coordinate Tokenization for Generative Visual Grounding

    Aug 4, 2026Xiuyuan Zhu, Ke Lu, Kun Dong +6Vision-Language Grounding