Visual Marking as a Native Reasoning Capability for Multimodal LLMs
Organizations: The Hong Kong University of Science and Technology
Abstract
Visual reasoning often requires maintaining intermediate spatial states that are difficult to express in text. Existing approaches either rely on external drawing tools or image generators, or encode these states in latent tokens without explicit spatial correspondence to the input image. We introduce SelfMark, which enables MLLMs to construct and use visual marks directly in their native visual feature space. By treating the visual token grid as a canvas, SelfMark establishes explicit spatial correspondence between marks and images. This is achieved through a locate-mark-read procedure, where the model identifies relevant locations, constructs task-dependent mark features, and reads the resulting visual features to guide its reasoning. We jointly train a single model on spatial planning, dense counting, high-resolution VQA, and line tracing. Across six benchmarks, SelfMark outperforms state-of-the-art MLLM-based methods specialized for individual tasks. These improvements also persist with a stronger backbone, demonstrating the effectiveness of visual marking across different model capabilities. Overall, SelfMark provides a unified approach to visual reasoning by integrating spatially grounded marks directly into the visual feature space of MLLMs.
Figures & tables
| Rely on | Tools / image generation | Standard MLLM |
|---|---|---|
| Spatial correspondence | Tools : DeepEyes [ 56 ] , Pixel-Reasoner [ 41 ] , Mini-o3 [ 18 ] , ZoomEye [ 34 ] Unified models : MVoT [ 21 ] , MathCanvas [ 35 ] | SelfMark (ours) : grid-aligned marks in visual feature space |
| Abstract representation | VisProg [ 15 ] , ViperGPT [ 36 ] , MM-REACT [ 50 ] | Text : VL-Rethinker [ 40 ] , WS-COC [ 55 ] Latent : Mirage [ 51 ] , CoLVR [ 10 ] , LatentGeo [ 48 ] |
| Type | Task | Condition per sub-cell | Loss |
|---|---|---|---|
| Points | counting | expected objects | Poisson NLL |
| Paths | spatial planning | route coverage | BCE |
| Boxes | high-res. VQA | region coverage | BCE |
| Lines | line tracing | curve coverage | BCE |
| Map size | |||||
| Method | Avg. | ||||
| Zero-shot MLLMs | |||||
| Qwen2.5-VL-7B [ 4 ] | 13.0 | 8.0 | 7.0 | 2.0 | 7.5 |
| Qwen2.5-VL-32B [ 4 ] | 28.0 | 23.0 | 14.0 | 11.0 | 19.0 |
| InternVL3.5-8B [ 44 ] | 37.0 | 26.0 | 11.0 | 2.0 | 19.0 |
| LLaVA-OV-7B [ 19 ] | 6.0 | 6.0 | 3.0 | 0.0 | 3.8 |
| VSP | FSC-147 | V* | HR-Bench | Tangled | ||
| Acc. | MAE | Acc. | 4K | 8K | Acc. | |
| Qwen2.5-VL-7B | ||||||
| LoRA baseline | 78.0 | 17.87 | 80.8 | 71.8 | 63.7 | 13.9 |
| SelfMark | 92.9 | 14.44 | 86.1 | 75.1 | 71.4 | 50.3 |
| Qwen3.5-9B | ||||||
| LoRA baseline | 90.5 | 14.12 | 86.1 | 73.8 | 67.1 | 71.8 |
| VSP | FSC-147 | V* | HR-Bench | Tangled | ||
| Marks | Acc. | MAE | Acc. | 4K | 8K | Acc. |
| Predicted | 92.9 | 14.44 | 86.1 | 75.1 | 71.4 | 50.3 |
| None | 76.8 | 16.74 | 80.3 | 68.9 | 64.8 | 13.0 |
| Mirrored | 76.3 | 14.41 | 79.4 | 73.0 | 64.3 | 17.4 |
| Ground Truth | 99.1 | 6.35 | 93.9 | – | – | 99.8 |
| w/o marking | 72.6 | 15.00 | 79.3 | 71.1 | 63.0 | 46.0 |
| Qwen2.5-VL-7B | Qwen3.5-9B | |
| Model | ||
| Input size | ||
| Locate input size † | ||
| LoRA | 2 adapters, rank 16, , all linear layers | |
| Editor | 8 blocks, width 1,536, 12 heads | |
| Locate heads | two-layer MLP of width 1,024 per mark type | |
| Type | Task | Locate input | Condition per sub-cell | Label | Drawn for | Loss |
|---|---|---|---|---|---|---|
| Points, <PTS> | counting | expected objects | annotated points | dot, radius 2 px | Poisson NLL | |
| Paths, <PATH> | spatial planning | route coverage | squares of all shortest routes | square | BCE | |
| Boxes, <BOX> | high-res. VQA | region coverage | region boxes (Sec. S2.2 ) | none, read as a crop | BCE | |
| Lines, <LINE> | line tracing | curve coverage | curve from the asked box | stroke, 12 px wide | BCE |
| Method | Training data |
|---|---|
| Spatial planning | |
| Mirage [ 51 ] | 1,000 VSP maps with helper images |
| CoLVR [ 10 ] | the 1,000 maps of Mirage and 500 maps of its RL stage |
| SelfMark | the maps of CoLVR, 1,497 after removing duplicates |
| Dense counting | |
| WS-COC [ 55 ] | 3,659 FSC-147 training images with counts |
| Method | Accuracy |
|---|---|
| Reported by KLAL [ 12 ] | |
| GPT-4o / Gemini-2.0 Flash | 55.3 / 59.3 |
| Qwen2.5-VL-7B, zero-shot / fine-tuned / with KLAL | 49.6 / 53.8 / 62.2 |
| Ours, Qwen2.5-VL-7B | |
| LoRA baseline | 96.9 |
| SelfMark / without marks | 96.3 / 96.2 |
| Benchmark | Items | Decoding | Answer |
|---|---|---|---|
| VSP | 400 | greedy | boxed plan, simulated |
| FSC-147 | 1,286 / 1,190 | greedy | first number † |
| V* | 191 | Avg@32 | option letter |
| HR-Bench 4K / 8K | 800 / 800 | Avg@8 | option letter |
| Tangled Lines | 600 | greedy | boxed letter |
| Evaluation | V* | HR 4K | HR 8K | Prior evaluation |
| DeepEyes | ||||
| Original script | 86.9 | – | – | 89.0 [ 38 ] |
| Shuffled options | 85.3 | – | – | 84.3–85.9 [ 38 ] |
| Released questions | 83.0 | 73.3 | 69.7 | 83.3 / 73.2 / 69.5 [ 18 ] |
| Pixel-Reasoner | ||||
| Released questions | 85.9 | 72.4 | 64.9 | 84.3 [ 41 ] |
| Method | V* | HR-8K | FSC-147 | VSP | Tangled |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B | 0.46 | 2.08 | 0.09 | 6.93 | 0.10 |
| Qwen3-VL-8B-Thinking | 5.07 | 21.66 | 72.65 | 189.11 | 27.20 |
| DeepEyes | 3.83 | 8.93 | – | – | – |
| Pixel-Reasoner | 4.86 | 5.00 | – | – | – |
| WS-COC | – | – | 0.17 | – | – |
| LoRA baseline | 0.51 | 0.72 | 0.19 | 0.39 | 0.28 |
VLM Reasoning
Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.
See all 875 papers