cs.CVMar 30, 2026
SaveMolmoPoint: Better Pointing for VLMs with Grounding Tokens
Organizations: Allen Institute for AI · University of Washington
Abstract
Grounding has become a fundamental capability of vision-language models (VLMs). Most existing VLMs point by generating coordinates as part of their text output, which requires learning a complicated coordinate system and results in a high token count. Instead, we propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept. Our model generates a special pointing token that cross-attends to the input image or video tokens and selects the appropriate one. To make this model more fine-grained, we follow these pointing tokens with an additional special token that selects a fine-grained subpatch within the initially selected region, and then a third token that specifies a location within that subpatch. We further show that performance improves by generating points sequentially in a consistent order, encoding the relative position of the previously selected point, and including a special no-more-points class when selecting visual tokens. Using this method, we set a new state-of-the-art on image pointing (70.7% on PointBench), set a new state-of-the-art among fully open models on GUI pointing (61.1% on ScreenSpotPro), and improve video pointing (59.1% human preference win rate vs. a text coordinate baseline) and tracking (+6.3% gain on Molmo2Track). We additionally show that our method achieves much higher sample efficiency and discuss the qualitative differences that emerge from this design change.
Figures & tables
Figure 1: Overview of MolmoPoint . To point, our model scores coarse-grained image patches using the LLM’s hidden states, then scores fine-grained subpatches from the highest scoring patch using ViT image features, and then selects a point within the highest scoring subpatch.
Figure 2: Pointing with grounding tokens . Keys are built from image tokens and ViT patch features, and queries are built from the <PATCH> token and <SUBPATCH> token hidden states, to score patches and subpatches. The <LOCATION> token predicts the final output points within the highest scoring subpatch.
Figure 3: Overview of the generation of MolmoPoint-GUISyn. We prompt an LLM to generate the HTML for the screenshot and extract all bounding boxes of its UI elements. Then we use LLMs to annotate each bounding box with its interaction intents.
Figure 4: MolmoPoint-TrackAny: human-annotated point-to-track extension. Annotators are given a text query and an object of interest, and provide point tracks while marking frames as occluded when the object is not visible.
| Model | Aff. | Spat. | Reason | Steer. | Count. | Avg |
| Human | 92.3 | 83.6 | 87.8 | 86.3 | 95.6 | 89.1 |
| API call only | ||||||
| Gemini-Robotics-ER-1.5 [ 1 ] | 69.7 | 69.7 | 60.1 | 67.5 | 68.5 | 67.1 |
| Gemini-2.5-Pro [ 20 ] | 72.7 | 70.3 | 71.0 | 41.0 | 59.2 | 62.8 |
| Open weights | ||||||
| Poivre-7B [ 87 ] | - | - | - | - | - | 67.5 |
Table 1: Point-Bench results. Baseline scores taken from the Point-Bench leaderboard. Qwen3-VL-235B-A22B-Instruct and VisionReasoner-7B scores were taken from Poivre [ 87 ] , which did not include sub-category scores.
| Metric | API-only | Open-weights | Fully-open | Ours | |||||||
| GPT-5.2 | Gemini3-Pro | Qwen3-VL-8B | Qwen3-VL-4B | Molmo-7B-D | Molmo-72B | Molmo-7B-O | Molmo2-4B | Molmo2-8B | Molmo2-O-7B | MolmoPoint-8B | |
| Recall | 31.0 | 77.3 | 54.3 | 45.1 | 76.4 | 74.9 | 74.4 | 83.3 | 85.5 | 83.1 | 90.4 |
| Precision | 32.9 | 81.3 | 53.5 | 44.2 | 76.2 | 74.9 | 74.6 | 85.1 | 86.4 | 83.7 | 89.3 |
| F1 | 31.6 | 77.8 | 53.4 | 43.7 | 75.7 | 74.5 | 74.0 | 83.4 | 85.2 | 82.7 | 89.2 |
Table 2: PixMo-Points results. MolmoPoint-8B surpasses even proprietary models. We collect results for GPT-5.2, Gemini-3, and Qwen3-VL ourselves.
| Model | ScreenSpot-V2 | ScreenSpot-Pro | OSWorldG |
| API call only | |||
| Claude 3.7 [ 3 ] | 87.6 | 27.7 | - |
| OpenAI CUA [ 55 ] | 87.9 | 23.4 | - |
| Gemini-3-Pro [ 28 ] | 93.7 | 72.7 | 35.5 |
| Open weights | |||
| Holo2-8B [ 21 ] | 93.2 | 58.9 | 70.1 |
Table 3: GUI grounding results. MolmoPoint-GUI-8B is our GUI specialized model finetuned on MolmoPoint-GUISyn (Synthetic GUI dataset) we constructed. Molmo2-GUI-8B also fine-tunes on MolmoPoint-GUISyn but without our token referencing mechanism. (64crops) denotes our test-time scaling for inference with more image crops. The best performance of fully open models is bold . Scores with * are from evaluations in [ 69 , 58 ] .
| BURST VC (test) [ 5 ] | Molmo2-VC | Molmo2-VP | |||||
| Model | Acc. | Close acc. | Acc. | Close acc. | F1 | Recall | Precision |
| API call only | |||||||
| GPT-5 [ 56 ] | 43.1 | 73.7 | 35.8 | 50.3 | 4.1 | 4.4 | 4.2 |
| GPT-5 mini [ 56 ] | 46.0 | 73.0 | 29.8 | 49.3 | 2.2 | 2.2 | 2.2 |
| Gemini 3 Pro [ 28 ] | 44.0 | 71.7 | 37.1 | 53.1 | 20.0 | 27.4 | 19.8 |
| Gemini 2.5 Pro [ 20 ] | 41.6 | 70.0 | 35.8 | 56.5 | 13.0 | 14.5 | 13.6 |
Table 4: Video counting and pointing results. MolmoPoint-8B scores highest on BURST-VC and MolmoPoint-8B-VP and second highest on MolmoPoint-8B-VC’s close accuracy, slightly behind Gemini 2.5 Pro. Best open model results are bold .
| MeViS [ 23 ] | MeViS [ 23 ] | Ref-YT-VOS [ 61 ] | Ref-Davis [ 34 ] | ReasonVOS [ 7 ] | |||||||||
| valid | valid-u | valid | valid | test | |||||||||
| Model | F1 | HOTA | F1 | HOTA | F1 | HOTA | F1 | HOTA | |||||
| API call only | |||||||||||||
| GPT-5 [ 56 ] | 23.4 | 26.5 | 17.3 | 14.0 | 30.9 | 21.0 | 18.4 | 25.2 | 17.0 | 11.6 | 24.7 | 13.6 | 10.7 |
| GPT-5 mini [ 56 ] | 15.7 | 15.4 | 8.5 | 6.8 | 16.2 | 7.4 | 6.2 | 8.4 | 3.4 | 2.3 | 14.6 | 4.2 | 3.4 |
| Gemini 3 Pro [ 28 ] | 42.5 | 51.1 | 42.3 | 36.0 | 55.0 | 49.1 | 45.5 | 66.6 | 60.8 | 55.7 | 52.6 | 48.5 | 42.1 |
Table 5: Tracking Results on Academic Benchmark . measures the segmentation mask quality of object tracks. F1 measures whether points fall in the mask, and HOTA [ 48 ] further accounts for consistent ID associations.
| Animals | Person | Sports | Dancers | Misc | Overall | |||||||||||||
| Model | F1 | HOTA | F1 | HOTA | F1 | HOTA | F1 | HOTA | F1 | HOTA | F1 | HOTA | ||||||
| API call only | ||||||||||||||||||
| GPT-5 [ 56 ] | 41.4 | 20.6 | 20.3 | 16.5 | 4.5 | 4.2 | 14.4 | 2.0 | 2.5 | 33.8 | 11.7 | 11.5 | 14.6 | 2.2 | 1.6 | 23.5 | 7.5 | 7.5 |
| GPT-5 mini [ 56 ] | 21.7 | 7.8 | 8.0 | 8.6 | 1.6 | 1.5 | 10.7 | 0.6 | 0.8 | 15.6 | 2.1 | 2.0 | 13.5 | 0.6 | 0.4 | 12.7 | 2.1 | 2.1 |
| Gemini 3 Pro [ 20 ] | 70.4 | 62.3 | 60.0 | 44.5 | 30.7 | 29.2 | 23.4 | 10.3 | 8.8 | 55.6 | 44.3 | 37.8 | 35.3 | 18.3 | 14.4 | 44.6 | 32.2 | 29.1 |
| Gemini 2.5 Pro [ 20 ] | 69.3 | 56.8 | 53.2 | 50.0 | 33.6 | 31.9 | 29.7 | 10.8 | 8.9 | 55.9 | 39.4 | 32.2 | 34.7 | 17.6 | 18.3 | 47.9 | 31.2 | 27.8 |
Table 6: Results on Molmo2Track benchmark by video domain . Overall reports the accuracy across all samples.
| Model | Animals | Person | Sports | Dancers | Misc | Overall | ||||||
| F1 | HOTA | F1 | HOTA | F1 | HOTA | F1 | HOTA | F1 | HOTA | F1 | HOTA | |
| MolmoPoint-8B-Ablation | 83.4 | 85.3 | 49.2 | 49.1 | 56.5 | 57.6 | 64.9 | 63.7 | 47.5 | 44.5 | 59.3 | 59.3 |
| w/o grounding tokens | 80.7 | 81.8 | 44.4 | 44.5 | 49.4 | 51.4 | 62.9 | 62.9 | 37.9 | 35.5 | 54.7 | 55.3 |
| w/o MolmoPoint-Track | 80.0 | 81.0 | 43.5 | 43.8 | 49.4 | 50.5 | 61.8 | 61.6 | 30.3 | 28.1 | 53.8 | 54.2 |
Table 7: Tracking ablations on Molmo2Track . We consecutively remove grounding tokens and the newly introduced MolmoPoint-Track data.
| Model | Image Avg. | Multi-Image Avg. | Short Video Avg. | Long Video Avg. |
| API call only | ||||
| GPT-5 [ 56 ] | 83.7 | 72.1 | 73.1 | 76.3 |
| GPT-5 mini [ 56 ] | 81.9 | 68.2 | 66.8 | 69.8 |
| Gemini 3 Pro [ 28 ] | 86.2 | 81.9 | 71.0 | 78.8 |
| Gemini 2.5 Pro [ 20 ] | 81.3 | 72.4 | 71.1 | 80.4 |
| Gemini 2.5 Flash [ 20 ] | 79.3 | 68.3 | 67.0 | 74.5 |
Table 8: Image and video QA results . Averaged results on the QA benchmarks in [ 19 ] , see the appendix for details. Some results were computed by [ 19 ] .
| Model | PixMoPoint | PointBench | Molmo2-VC | ||
| F1 | Avg. | Correct | Close | Overcount | |
| Molmo2-P-Ablation | 85.2 | 67.8 | 58.0 | 36.6 | 3.6 |
| w/o rotary | 84.5 | 67.6 | 56.8 | 33.0 | 4.5 |
| w/o no-more-points | 84.7 | 66.6 | 52.3 | 32.8 | 10.3 |
| w/o point sorting | 83.6 | 71.2 | 40.0 | 24.4 | 3.2 |
Table 9: Ablations . Results when removing rotary embeddings, the no-more-points class, or the constraint that the points must be generated in sorted order.
Figure 5: Sample efficiency . Left: Performance when using a very limited number of pointing training examples. Right: Pointing performance during full-scale pre-training.
Figure 6: Qualitative examples . Top, Molmo2 generates lines of incorrect points in multiple video frames. Middle, Molmo2 is unable to localize the “x” exactly (zoomed images are close-ups of the same point). Bottom left, MolmoPoint-8B finds the second, partly occluded saddle. Bottom right, MolmoPoint-8B misses one of the plates. The middle row shows results from MolmoPoint-GUI-8B and Molmo2-GUI-8B, and the others show MolmoPoint-8B and Molmo2-8B.
| Dataset Group | MolmoPoint-8B | Molmo2 |
| Captioning/Long QA | 15.0 | 13.6 |
| Image QA | 25.0 | 22.7 |
| Video QA | 20.0 | 18.2 |
| Image Pointing | 7.0 | 9.1 |
| Video Pointing | 11.0 | 13.6 |
| Video Tracking | 12.0 | 13.6 |
Table 10: Updated top-level sampling rates . For MolmoPoint-8B we slightly reduce the sampling rates on pointing tasks since we observe faster convergence of these tasks. Each of these categories consists of its own sub-mixture of datasets. We leave the sampling ratios within those submixtures unchanged. See [ 19 ] for the full list.
| Model | phase | Hardware | GPUs | hours | GPU hr. | batch | steps | mm. | annotations |
| MolmoPoint-8B | Pretrain | B200 | 32 | 10.3 | 330 | 128 | 23k | 4.6m | 14m |
| MolmoPoint-8B | SFT | B200 | 80 | 86.4 | 6.9k | 160 | 22k | 13m | 49m |
| MolmoPoint-8B | SFT-LC | B200 | 80 | 36.2 | 2.9k | 160 | 2k | 2.2m | 7.5m |
| MolmoPoint-GUI-8B | SFT | B200 | 32 | 9.4 | 300 | 128 | 2k | 360k | 11m |
| MolmoPoint-Vid-8B | Pretrain | H100 | 32 | 10.1 | 323 | 128 | 23k | 3.5m | 6.6m |
| MolmoPoint-Vid-8B | SFT | B200 | 32 | 13.7 | 437 | 64 | 6k | 1.4m | 5.1m |
Table 11: Training times . Columns show the model, the training phase, gpu counts, training hours, total GPU hours, batch size, the estimated number of multi-modal inputs that were seen (separately counting inputs that were packed together), and the estimated number of training annotations that were seen (separately counting annotations that were merged into message trees).
| Model | NextQA test [ 79 ] | PerceptionTest test [ 57 ] | MVBench test [ 39 ] | Tomato test [ 62 ] | MotionBench val [ 31 ] | TempCompass test MCQ [ 43 ] | Video-MME test [ 25 ] | Video-MME-Sub test [ 25 ] | LongVideoBench val [ 75 ] | MLVU test MCQ [ 101 ] | LVBench test [ 72 ] | VideoEvalPro test [ 49 ] | Ego Schema test [ 50 ] | Molmo2 Caption test F1 Score | Molmo2 Count val accuracy | Short QA avg. | Long QA avg. | Average |
| API call only | ||||||||||||||||||
| GPT-5 [ 56 ] | 86.3 | 79.4 | 74.1 | 53.0 | 65.4 | 80.4 | 83.3 | 86.9 | 72.6 | 77.7 | 65.2 | 68.8 | 75.6 | 50.1 | 35.8 | 73.1 | 76.3 | 70.6 |
| GPT-5 mini [ 56 ] | 83.2 | 72.0 | 66.5 | 44.1 | 59.9 | 74.9 | 77.3 | 82.3 | 69.7 | 69.1 | 54.7 | 60.1 | 70.9 | 56.6 | 29.8 | 66.8 | 69.8 | 65.0 |
| Gemini 3 Pro [ 28 ] | 84.3 | 77.6 | 70.4 | 48.3 | 62.6 | 82.8 | 88.6 | 87.5 | 75.9 | 75.7 | 77.0 | 78.0 | 68.9 | 36.0 | 37.1 | 71.0 | 78.8 | 70.0 |
| Gemini 2.5 Pro [ 20 ] | 85.3 | 78.4 | 70.6 | 48.6 | 62.0 | 81.9 | 87.8 | 87.8 | 76.8 | 81.5 | 75.7 | 78.4 | 72.2 | 42.1 | 35.8 | 71.1 | 80.4 | 71.2 |
| Gemini 2.5 Flash [ 20 ] | 81.8 | 74.7 | 67.0 | 39.1 | 59.3 | 80.2 | 84.2 | 84.2 | 73.1 | 75.1 | 64.9 | 69.6 | 70.2 | 46.0 | 31.9 | 67.0 | 74.5 | 66.7 |
Table 12: Video benchmark results for a range of proprietary APIs, open-weight baselines, fully-open baselines, Molmo2, and MolmoPoint-8B across video understanding, captioning, and counting benchmarks. The best-performing open-weight model is in bold , and the second-best is underlined . For MolmoPoint-8B, we report PerceptionTest val since the test server is no longer operational.
| Model | AI2D test [ 33 ] | ChartQA test [ 51 ] | DocVQA test [ 53 ] | InfoQA test [ 52 ] | TextVQA val [ 63 ] | VQA v2.0 val [ 30 ] | RWQA [ 78 ] | MMMU val [ 95 ] | MathVista testmini [ 47 ] | CountBench [ 8 ] | PixMoCount test [ 22 ] | MuirBench [ 70 ] | MMIU [ 54 ] | Blink val [ 26 ] | Img QA avg. | MultiImg QA avg. | Average |
| API call only | |||||||||||||||||
| GPT-5 [ 56 ] | 89.5 | 83.8 | 88.9 | 83.0 | 78.7 | 79.7 | 80.8 | 81.8 | 82.7 | 90.8 | 67.2 | 78.6 | 71.0 | 66.5 | 82.5 | 72.1 | 80.2 |
| GPT-5 mini [ 56 ] | 86.7 | 82.1 | 86.7 | 82.2 | 79.1 | 72.1 | 77.0 | 78.7 | 79.2 | 87.1 | 74.4 | 71.4 | 64.5 | 68.7 | 80.5 | 68.2 | 77.8 |
| Gemini 2.5 Pro [ 20 ] | 94.3 | 77.8 | 91.5 | 82.0 | 70.3 | 67.1 | 77.4 | 79.6 | 84.6 | 90.8 | 73.8 | 74.5 | 68.9 | 73.7 | 80.8 | 72.4 | 79.0 |
| Gemini 2.5 Flash [ 20 ] | 95.9 | 76.8 | 91.1 | 80.9 | 73.0 | 69.4 | 74.5 | 79.0 | 81.2 | 86.7 | 63.9 | 73.5 | 61.2 | 70.2 | 79.3 | 68.3 | 76.9 |
| Claude Sonnet 4.5 [ 4 ] | 91.5 | 80.2 | 91.7 | 65.9 | 67.2 | 77.0 | 61.1 | 77.8 | 73.1 | 87.3 | 58.3 | 59.6 | 54.1 | 64.8 | 75.6 | 59.5 | 72.1 |
Table 13: Image benchmark results for a range of proprietary APIs, open-weight baselines, and MolmoPoint-8B across a range of image understanding and counting benchmarks. The result of the best-performing open-weight model is in bold , and the second best is underlined .
Figure 7: Examples from MolmoPoint-TrackSyn data. Object tracks, text queries, and videos are generated synthetically with static and dynamic camera viewpoints.
Figure 8: Qualitative examples of MolmoPoint-GUISyn. We demonstrate GUI grounding examples for desktop, web, and mobile screenshots.
More in this topic
Vision-Language Grounding
Vision-Language Grounding links words and phrases to the objects, regions, actions, or moments they describe in images, video, and scenes.
See all 418 papers