cs.CVNov 8, 2026

LocateAnyMesh: Grounding 3D Mesh Queries in 2D Images with Vision-Language Models

Authors: Chris Choy, Zhiding Yu, Jan Kautz

Organizations: NVIDIA Research

Abstract

Textured 3D meshes specify object geometry and appearance, making them a natural query for finding a particular asset in an image. We introduce LocateAnyMesh, a vision-language model that uses the mesh itself as part of its prompt. A frozen 3D encoder maps colored surface points to a compact token sequence, which a learned connector places alongside image and instruction tokens. We evaluate two grounding models with PPAT mesh tokens: LAM-Select selects image-region proposals, while LAM-Direct predicts coordinates directly. Neither requires rendered query views or view-template matching. New objects require a single encoder pass and no object-specific fine-tuning. LAM-Select improves localization on the reported real-image suites and the official BOP YCB-V evaluation. Secondary experiments compare mesh encoders, and a LAM-Direct study isolates surface color. A single-seed LAM-Select continuation supplies pointing and mesh-only description evidence without replacing the reported localization model. The results support mesh tokens as a query representation, with remaining limitations in presence decisions, identity binding, proposal coverage, and descriptive fidelity.

Figures & tables

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes

    Jun 30, 2026Kartik Bali, Roland AydinOpen-Vocabulary 3D Segmentation3D VLMs

  2. AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models

    May 25, 2026Cuong Huynh, Maxim Popov, Denis Gridusov +13D Visual Grounding

  3. 3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models

    Jun 18, 2026Jintang Xue, Xinyu Wang, Yixing Wu +23D Representation LearningMultimodal Foundation Models