LocateAnyMesh: Grounding 3D Mesh Queries in 2D Images with Vision-Language Models
Organizations: NVIDIA Research
Abstract
Textured 3D meshes specify object geometry and appearance, making them a natural query for finding a particular asset in an image. We introduce LocateAnyMesh, a vision-language model that uses the mesh itself as part of its prompt. A frozen 3D encoder maps colored surface points to a compact token sequence, which a learned connector places alongside image and instruction tokens. We evaluate two grounding models with PPAT mesh tokens: LAM-Select selects image-region proposals, while LAM-Direct predicts coordinates directly. Neither requires rendered query views or view-template matching. New objects require a single encoder pass and no object-specific fine-tuning. LAM-Select improves localization on the reported real-image suites and the official BOP YCB-V evaluation. Secondary experiments compare mesh encoders, and a LAM-Direct study isolates surface color. A single-seed LAM-Select continuation supplies pointing and mesh-only description evidence without replacing the reported localization model. The results support mesh tokens as a query representation, with remaining limitations in presence decisions, identity binding, proposal coverage, and descriptive fidelity.
Figures & tables
| Variant | Base model | LLM | Params. | Vision encoder(s) | Output | External detector at eval. |
| LAM -Direct | LocateAnything-3B | Qwen2.5-3B-Instruct | 3B class | MoonViT-SO-400M | Box coordinates | None |
| LAM -Select | VLX-Seek-1.5-10B | Qwen3.5 | 10B class | Qwen3.5-VL + C-RADIO | Region references | WeDetect-Base-Uni, top 100 |
| System | HOPE | LM-O | HB |
| LAM -Direct | 0.155 | 0.181 | 0.303 |
| LAM -Select | 0.371 | 0.388 | 0.366 |
| Grounding system | AP | AP50 | AP75 | GB200 s/img , | Repeat range s/img |
| LAM -Direct | 0.6384 | 0.7885 | 0.7112 | 0.322 | 0.309–0.324 |
| LAM -Select | 0.7152 | 0.9150 | 0.8183 | 0.682 | 0.652–0.837 |
| Outcome | Instances / queries |
| Matched target instances | 747 |
| No matching proposal | 68 |
| Covered, but refused | 470 |
| Covered, but selection missed | 196 |
| Absent-query false positives | 311 |
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Source | Prepared query rows |
| GSO v2 | 1,106,785 |
| ABO | 978,042 |
| YCB-V PBR | 703,878 |
| Objaverse PPAT union | 627,079 |
| YCB-V real | 622,876 |
| Objaverse tabletop v2tex | 73,784 |
| LAM -Select + PPAT | LAM -Direct + PPAT | |
| Added tokens | <MESH_CONTEXT> (id 248182) | <MESH_CONTEXT> , <mesh> , </mesh> (ids 152681–152683) |
| Mesh span | <MESH_CONTEXT> positions | <mesh> , <MESH_CONTEXT> positions, </mesh> |
| Visual input | Native image tokens and one <objN><objfeat> pair per candidate | Inherited <img> , <IMG_CONTEXT> tokens |
| Output | <ground>mesh</ground> then <objects> enclosing region IDs, followed by </objects> | <box><x1><y1><x2><y2></box> , integer coordinates in |
| Multiple boxes | Multiple <objN> references within the object span | Concatenated <box>...</box> blocks; main benchmark caps output at one box |
| Absent | None inside the object span | <box>none</box> ; plain-text none , id 6697 |
| LAM -Select + PPAT | LAM -Direct + PPAT (arm A) | |
| Initialization | VLX-Seek-1.5-10B | LocateAnything-3B |
| Text decoder | Qwen3.5, layers, width | Qwen2.5, layers, width |
| Frozen mesh features | PPAT, tokens of width | |
| Connector widths | ||
| LoRA / / dropout | 16 / 32 / 0 | 64 / 128 / 0.05 |
| LoRA targets | All language-model linear layers | + gate, up, down |
| Source | Objects pool/use | Rows | Rep. | Effective rows (share) |
| ycbv_train_real | 21/21 | 623,479 | 1.0 | 623,479 (14.90%) |
| ycbv_train_pbr | 21/21 | 702,649 | 1.0 | 702,649 (16.79%) |
| gso_v2 | 971/874 | 00 5,080 | 218.0 | 1,107,440 (26.46%) |
| abo | 4,047/3,642 | 00 2,397 | 408.0 | 0 977,976 (23.36%) |
| objv2tex | 11,909/11,909 | 109,043 | 3.40 | 0 371,022 0 (8.86%) |
| objv_wave2 | 11,909/11,909 | 118,505 | 3.40 | 0 403,216 0 (9.63%) |
| Suite | Split | Obj. | Scenes | Images eval. | Present queries | LAM -Select + PPAT | LAM -Direct + PPAT |
| YCB-V BOP | official targets | 21 | 12 | 900 | 4,123 | ✓ | ✓ |
| LM-O | lmo/test | 8 | 1 | 300 | 2,319 | ✓ | ✓ |
| HOPE | hope/val | 28 | 10 | 50 | 0 775 | ✓ | ✓ |
| HB | hb/val_prim. | 33 | 13 | 300 | 1,553 (2,400 abs.) | ✓ | ✓ |
| GSO/ABO/Objv. h/o | renders | 97 / 405 / 385 | New A | 300 | New A | n/e | ✓ |
| Outcome | PPAT | Uni3D-S |
| Matched target instances | 747 | 826 |
| No matching proposal | 68 | 68 |
| Covered, but refused | 470 | 406 |
| Covered, but selection missed | 196 | 181 |
| Absent-query false positives | 311 | 352 |
| Suite | Encoder | Matched | Uncovered | Covered refused | Covered missed | Absent FP / queries |
| HOPE | PPAT | 600 | 0 | 26 | 294 | 245 /400 |
| Uni3D-S | 577 | 0 | 83 | 260 | 272/400 | |
| LM-O | PPAT | 1,414 | 249 | 415 | 241 | 11 /81 |
| Uni3D-S | 1,241 | 249 | 528 | 301 | 25/81 |
| Co-visible donor / 18 | Absent donor / 18 | |||
| System | Primary | Exact set | Primary | Exact set |
| LAM -Direct + PPAT | 1 | 0 | 9 | 9 |
| LAM -Select + PPAT | 5 | 3 | 11 | 9 |
| LAM -Select + Uni3D-S | 4 | 3 | 10 | 9 |
| System | GSO | ABO | Objaverse |
| LAM -Direct + PPAT | 0.305 | 0.353 | 0.320 |
| System | Encoder | AP | AP50 | AP75 | GB200 s/img | Repeat range s/img |
| LAM -Direct | PPAT | 0.6384 | 0.7885 | 0.7112 | 0.322 | 0.309–0.324 |
| LAM -Select | PPAT | 0.7152 | 0.9150 | 0.8183 | 0.682 | 0.652–0.837 |
| LAM -Select | Uni3D-S | 0.7134 | 0.9110 | 0.8162 | 0.882 | 0.705–0.921 |
| Mesh encoder | HOPE | LM-O | HB (all) | HB (no dev.) |
| PPAT | 0.3707 | 0.3880 | 0.3658 | 0.3404 |
| Uni3D-S | 0.3160 | 0.3337 | 0.4559 | 0.4219 |
| Prompt | Cap | Recall | Precision | Both found / 48 | Single exact / 8 | Absent FP / 8 |
| Single | 1 | 24.0 | 48.9 | 0 | 4 | 7 |
| Single | 4 | 29.2 | 21.9 | 4 | 2 | 7 |
| Every | 4 | 41.7 | 22.2 | 10 | 0 | 7 |
| System | Encoder | Queries/s | Queries/s | Approx. s/img | Peak alloc. GiB |
| LAM -Direct | PPAT | 7.89 | 65.3 | 0.322 | 7.9–8.5 |
| LAM -Direct | Uni3D-S | 7.70 | 59.9 | 0.351 | 7.9–8.4 |
| LAM -Select | PPAT | 2.07 | 30.8 | 0.682 | 21.4 |
| LAM -Select | Uni3D-S | 1.57 | 23.8 | 0.882 | 21.4 |
| System | Encoder | range | |||||
| LAM -Direct | PPAT | 7.89 | 21.5 | 35.8 | 44.2 | 65.3 | 64.9–67.9 |
| LAM -Direct | Uni3D-S | 7.70 | 20.4 | 33.9 | 44.4 | 59.9 | 52.7–63.4 |
| LAM -Select | PPAT | 2.07 | 6.86 | 12.5 | 16.5 | 30.8 | 25.1–32.2 |
| LAM -Select | Uni3D-S | 1.57 | 5.45 | 9.9 | 14.1 | 23.8 | 22.8–29.8 |
| Query point cloud | LM-O | HOPE | HB | Discrim | GSO | ABO | Objv. |
| Color twin | 0.198 | 0.122 | 0.244 | 2.356 | 0.266 | 0.302 | 0.257 |
| Gray twin | 0.040 | 0.052 | 0.177 | 1.798 | 0.124 | 0.241 | 0.208 |
| gray vs. color |
| Encoder package | HB | LM-O | HOPE | commit | FP | Discrim |
| PPAT-B32 (anchor) | 0.282 | 0.140 | 0.117 | 0.637 | 0.210 | 3.030 |
| Uni3D-S | 0.265 | 0.271 | 0.150 | 0.700 | 0.271 | 2.585 |
| Uni3D-G (Y-up) | 0.304 | 0.139 | 0.166 | 0.603 | 0.208 | 2.901 |
| Agreement rating / 10 | Same-asset judgments | |||
| System / source | Correct mesh | Random swap | Correct mesh | Random swap |
| LAM -Direct + PPAT, separate multitask checkpoint | ||||
| ABO ( ) | 3.73 | 1.18 | 7/40 | 0/40 |
| GSO ( ) | 3.63 | 1.30 | 5/40 | 1/40 |
| Objaverse ( ) | 1.98 | 0.43 | 0/40 | 0/40 |
| All ( ) | 3.11 | 0.97 | 12/120 | 1/120 |
| Condition | Identity S / U / C | Refusal-like text | Appropriate abstention | |
| Original present mesh | 32 | 11 / 18 / 3 | 0 | New A |
| Other present mesh | 32 | 7 / 17 / 4 | 4 | New A |
| Absent image, same suite | 25 | New A | 16 | 13 |
| Absent image, cross suite | 7 | New A | 6 | 5 |
3D Visual Grounding
Three-dimensional visual grounding links language or visual queries to specific objects, parts, regions, and spatial relations in 3D scenes.
See all 148 papers