cs.CVMar 15, 2026
SaveStAR: Segment Anything Reasoner
Organizations: KAIST AI · University of Seoul
Abstract
As AI systems are being integrated more rapidly into diverse and complex real-world environments, the ability to perform holistic reasoning over an implicit query and an image to localize a target is becoming increasingly important. However, recent reasoning segmentation methods fail to sufficiently elicit the visual reasoning capabilities of the base mode. In this work, we present Segment Anything Reasoner (StAR), a comprehensive framework that refines the design space from multiple perspectives-including parameter-tuning scheme, reward functions, learning strategies and answer format-and achieves substantial improvements over recent baselines. In addition, for the first time, we successfully introduce parallel test-time scaling to the segmentation task, pushing the performance boundary even further. To extend the scope and depth of reasoning covered by existing benchmark, we also construct the ReasonSeg-X, which compactly defines reasoning types and includes samples that require deeper reasoning. Leveraging this dataset, we train StAR with a rollout-expanded selective-tuning approach to activate the base model's latent reasoning capabilities, and establish a rigorous benchmark for systematic, fine-grained evaluation of advanced methods. With only 5k training samples, StAR achieves significant gains over its base counterparts across extensive benchmarks, demonstrating that our method effectively brings dormant reasoning competence to the surface.
Figures & tables
Figure 1: We establish a reasoning segmentation benchmark, ReasonSeg-X , that demands a broad range of reasoning skills. Our dataset addresses goal-oriented reasoning and the ability to flexibly invoke world knowledge, while also covering complex relational reasoning and step-by-step reasoning capabilities. Our model, StAR (equipped with Qwen3-VL 32B [ 1 ] ), demonstrates remarkable performance across all these aspects (Our model’s prediction is shown as the green mask in the right image). Additional results are provided in the supplementary material.
Figure 2: ReasonSeg-R examples. We correct masks for cases that are mismatched to the query and also refine mask quality. In addition, we modify query expressions that may inadvertently include regions outside the mask ( e.g. , the video player monitor).
Figure 3: Overview of the StAR framework pipeline. We pinpoint and resolve reasoning bottlenecks from all perspectives of the existing reasoning segmentation framework (highlighted in red text). See text for details.
Figure 4: (a) We progressively retrofit all pillars of the standard RLVR framework (VisionReasoner) toward our performant and robust model (StAR), without introducing significant computational overhead. The foreground bars show performance on ReasonSeg-X test , while results on ReasonSeg-R are shown with gray bars. The performance values in parentheses are obtained with our majority voting strategy (Sec. 3.3 ). Gray-marked design decisions (y-axis label texts) indicate comparison-only variants that are not adopted. (b) The proposed RLVR design space better preserves and exploits the base MLLM’s diverse reasoning capabilities (VisionReasoner StAR stage-1). Moreover, Stage 2 training on the ReasonSeg-X train set with rollout-expanded selective-tuning (REST) effectively elicits the MLLM’s hidden reasoning potential, improving performance particularly on complex reasoning ( e.g. , C/MH). The resulting model, StAR, outperforms the baseline by a large margin across all reasoning types.
Figure 5: Motivation for the mask IoU reward. Left : Misalignment between the MLLM-level reward function and the task’s final objective potentially induces confusing learning signals. Right : Distinguishing between diverse mask defects via a fine-grained tiered mask reward promotes stable and stepwise optimization.
Figure 6: Left : Rollout-expanded selective-tuning (REST) substantially improves sample utilization. In particular, for hard samples, it develops the model’s intricate reasoning by exploring a vast space of reasoning paths. “Correct” is defined as IoU > 0.5, and results are reported with the Qwen3-VL 8B model. Right : REST effectively boosts performance on problems that require multi-step reasoning; moreover, its gains synergistically scale with the base model’s reasoning capability boundaries.
| Method | Base model (Size) | RS | RS-R | RS-X | RS-X test | ||||||||||||
| test | – | val | overall | P/F | C/KI | C/R | C/MH | ||||||||||
| gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | ||
| Supervised/Reinforcement Finetuning Methods | |||||||||||||||||
| LISA (ft) | Llama2 13B | 51.5 | 51.3 | 52.5 | 53.3 | 23.8 | 24.3 | 25.1 | 26.0 | 27.9 | 30.6 | 28.2 | 29.2 | 26.8 | 25.8 | 13.8 | 16.8 |
| SegLLM | LLaVA1.5 7B | 52.4 | 48.4 | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| READ (ft) | LLaVA1.5 13B | 62.2 | 62.8 | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
Table 1: Performance results on the ReasonSeg (RS) series. “MV” denotes our majority voting strategy adapted for segmentation. We compare our StAR against state-of-the-art baselines: LISA [ 25 ] , SegLLM [ 60 ] , READ [ 40 ] , CoReS [ 3 ] , RSVP [ 34 ] , CoPRS [ 35 ] , SAM-R1 [ 16 ] , DPAD [ 67 ] , SAM-Veteran [ 8 ] , SegCompass [ 36 ] , DR 2 Seg [ 14 ] , VisionReasoner [ 33 ] , Seg-ReSearch [ 28 ] , SAM 3 Agent [ 5 ] , and RSAgent [ 13 ] .
| Method | Base model (Size) | val | test | ||||||
| Obj & Part | Obj & Part | Obj | Part | ||||||
| gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | ||
| LISA [ 25 ] | Llama2 13B | 15.4 | 20.0 | 16.1 | 19.8 | 26.1 | 27.9 | 7.4 | 8.4 |
| LISA ∗ [ 25 ] | Llama2 13B | 22.3 | 33.4 | 23.0 | 29.2 | 40.2 | 45.2 | 10.7 | 16.4 |
| M 2 SA ∗ [ 19 ] | LLaVA 7B | 27.8 | 48.6 | 30.9 | 46.8 | 41.0 | 55.6 | 13.5 | 27.0 |
| M 2 SA ∗ [ 19 ] | Llama2 13B | 28.4 | 49.1 | 31.6 | 47.6 | 42.3 | 57.6 | 13.6 | 27.2 |
Table 2: Comparison on the Multi-target and Multi-granularity Reasoning (MMR) dataset. * indicates models trained on a training set that includes the MMR dataset.
| Success rate (%) | RefSpatial-Bench | |
| Method | Location | Unseen |
| Gemini-2.5-Pro | 46.96 | 27.14 |
| RoboRefer-8B [ 73 ] | 52.00 | 37.66 |
| StAR-8B | 62.00 | 38.96 |
Table 3: Robotics grounding performance on RefSpatial-Bench [ 73 ] .
Figure 7: Qualitative results on our ReasonSeg-X test set demonstrate that StAR models, via multi-step reasoning, can effectively tackle complex segmentation scenarios necessitating various capabilities, such as comparative/relational reasoning, physics-based 3D awareness, comprehensive pattern analysis, and multi-hop algebraic expansion.
Table 4: Left : We evaluate performance under varying degrees of REST exploration (the number of rollout generations ), focusing on the efficacy of the inference-time strategy ; the per-step training time for StAR-7B is reported below. Right : We find that reversing the prediction order at test time—after training the model to predict the label before bbox/point coordinates— causes a large performance degradation. Default settings are marked in blue .
Table S.1 : Prompt template for stage 1 training. “{ Question }” is replaced by user query.
Table S.2 : Prompt template for stage 2 training and inference. “{ Question }” is replaced by user query.
Figure S.1 : Examples of no-target samples in ReasonSeg-X. StAR-32B reliably identifies cases where the queried object does not exist in the image. The first row further shows that majority voting can mitigate several failure modes of one-shot prediction. Notably, the model exhibits diverse reasoning skills, such as fine-grained local inspection, sequential search over multiple constraints, and verification of intermediate conclusions.
Figure S.2 : We further illustrate typical refinement decisions. In the leftmost example, we decide to remove the sample because both the mask quality and the query’s required reasoning are visually and semantically ambiguous.
| Method | ReasonSeg-R | ReasonSeg-X test | ||||||
| gIoU | cIoU | #Tokens | Latency | gIoU | cIoU | #Tokens | Latency | |
| Measured on an NVIDIA RTX A6000 | ||||||||
| VisionReasoner [ 33 ] | 64.8 | 56.8 | 160 | 4.0s | 42.2 | 33.8 | 200 | 5.2s |
| StAR-7B | 69.7 | 66.2 | 138 | 3.5s | 49.2 | 43.6 | 147 | 3.7s |
| MV | 70.7 | 67.2 | 139 32 | 32.3s | 50.3 | 44.9 | 149 32 | 33.7s |
| StAR-8B | 73.8 | 69.2 | 203 | 2.9s | 57.9 | 50.1 | 267 | 3.7s |
Table S.3 : Model inference efficiency results. “MV” denotes our majority voting strategy. Results for SAM 3 Agent and majority voting are measured with batch size 1, while all other results are measured with batch size 32.
Table S.4 : Left: Performance comparison on the MUSE [ 45 ] dataset. In [ 45 ] , evaluation includes scores obtained by prompting GPT-3.5, which can hinder consistent and reproducible measurement in follow-up studies. We instead simply adopt a standard IoU calculation scheme to improve evaluation accessibility and focus on comparison with VisionReasoner [ 33 ] . Right: Performance comparison on Referring Expression Segmentation (RES) benchmarks measured by cIoU. * denotes our reproduced results with the official code and model checkpoint.
| Variant | RS-R | RS-X test | ||||||||||
| – | overall | P/F | C/KI | C/R | C/MH | |||||||
| gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | |
| VisionReasoner-7B [ 33 ] | 64.8 | 56.8 | 42.2 | 33.8 | 50.1 | 45.1 | 50.7 | 39.0 | 39.4 | 31.1 | 24.5 | 23.8 |
| FFT LoRA 16 | 64.0 | 58.0 | 44.4 | 34.9 | – | – | – | – | – | – | – | – |
| LoRA 16 64 | 66.1 | 58.0 | 45.4 | 37.7 | 54.9 | 49.7 | 55.7 | 46.9 | 42.9 | 35.9 | 22.4 | 21.6 |
| MLLM-level reward, rollout 8 16 | 65.5 | 55.9 | 45.6 | 37.2 | 55.5 | 44.9 | 54.0 | 42.9 | 43.0 | 35.2 | 25.1 | 25.5 |
Table S.5 : Detailed results for retrofitting a VisionReasoner [ 33 ] .
| StAR-7B | RS-R | RS-X test | ||||||||||
| – | overall | P/F | C/KI | C/R | C/MH | |||||||
| gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | |
| rank 16 | 68.9 | 64.2 | 46.8 | 40.2 | 54.4 | 49.3 | 57.1 | 47.7 | 44.3 | 37.3 | 27.0 | 26.8 |
| MV | 70.4 | 64.7 | 48.8 | 41.6 | 58.0 | 53.1 | 60.8 | 51.3 | 44.5 | 37.7 | 27.7 | 26.3 |
| rank 64 | 69.7 | 66.2 | 49.2 | 43.6 | 58.2 | 52.2 | 60.2 | 52.4 | 45.7 | 40.4 | 28.0 | 28.8 |
| MV | 70.7 | 67.2 | 50.3 | 44.9 | 60.6 | 56.9 | 61.8 | 54.4 | 46.2 | 40.1 | 27.9 | 30.5 |
Table S.6: Performance comparison between LoRA rank 16 vs. 64 on ReasonSeg-R/X.
| Mask Reward Function | RS-R | RS-X | RS-X test | |||
| – | val | overall | ||||
| gIoU | cIoU | gIoU | cIoU | gIoU | cIoU | |
| SAM-R1’s [ 16 ] | 69.3 | 65.3 | 49.6 | 43.2 | 48.1 | 42.6 |
| Ours | 69.7 | 66.2 | 50.5 | 48.0 | 49.2 | 43.6 |
Table S.7: Performance comparison with different mask-accuracy reward functions. Our reward design facilitates stable policy updates by finely distinguishing trajectory’s advantages.
Figure S.3 : Reasoning path comparison with and without REST.
Figure S.4 : Label prediction examples.
| Dataset | gIoU/cIoU (%) Model | Majority Voting (MV) IoU threshold | ||||
| 0.70 | 0.75 | 0.80 | 0.85 | 0.90 | ||
| RS-X | StAR-7B | 50.8/45.0 | 50.8/44.9 | 50.6/44.7 | 50.3/44.9 | 50.0/44.6 |
| StAR-8B | 59.3/52.8 | 59.5/52.8 | 59.5/53.0 | 59.6/53.1 | 59.5/52.9 | |
| RS-R | StAR-7B | 70.7/67.6 | 70.6/67.0 | 70.8/67.1 | 70.7/67.2 | 70.9/66.5 |
| StAR-8B | 75.0/69.4 | 75.0/69.5 | 74.9/69.4 | 74.9/69.7 | 74.9/69.7 | |
Table S.8 : MV performance under different IoU thresholds.
Figure S.5 : Our majority voting strategy is robust to various hyperparameters and improves performance across base model scales.
Figure S.6 : Visual comparison results on our ReasonSeg-X test set (P/F and C/KI).
Figure S.7 : Visual comparison results on our ReasonSeg-X test set (C/R and C/MH).
Figure S.8 : Qualitative results of StAR-8B/32B on our ReasonSeg-X (P/F and C/KI).
Figure S.9 : Qualitative results of StAR-8B/32B on our ReasonSeg-X (C/R).
Figure S.10 : Qualitative results of StAR-8B/32B on our ReasonSeg-X (C/MH).
Figure S.11 : Failure cases of our model on challenging ReasonSeg-X test samples. Correct reasoning steps and reasoning errors are highlighted in green and red, respectively. Best viewed when zoomed in.
More in this topic
Reasoning Segmentation
Reasoning segmentation links language and visual reasoning to pixel-level masks for targets defined by complex intent, relations, or context.
See all 34 papers