cs.CVOct 21, 2026
SaveLLMSeg: Training-Free Text-Prompted Image Segmentation with B-Splines
Organizations: King Abdullah University of Science and Technology (KAUST)
Abstract
Promptable segmentation models such as SAM 3 achieve remarkable open-vocabulary performance but require dedicated architectures, large-scale annotation engines, and task-specific training. We ask a different question: can a general-purpose multimodal large language model (MLLM) segment images directly, without an added segmentation head, mask decoder, or training? We present LLMSeg, a training-free pipeline that turns a vision-capable LLM into a text-prompted image segmenter. LLMSeg decomposes segmentation into three LLM-driven stages. First, detect localizes all instances matching a free-form text query and describes their coarse morphology. Next, segment traces each instance using closed periodic B-splines confined to its detected box. These splines define an outer boundary and optional hole loops. Finally, refine iteratively proposes focus boxes around residual errors and retraces confined splines. The process ends when the mask stabilizes or the pass budget is exhausted. The output is a binary mask with inspectable intermediate geometry. We evaluate LLMSeg across ten referring and reasoning segmentation splits (ReasonSeg and RefCOCO/+/g). We also evaluate open-vocabulary instance segmentation (LVIS, COCO, and SA-Co), semantic segmentation, and zero-shot detection (ODinW13 and RF-100VL). With an off-the-shelf Gemini-3.7-Flash backend, LLMSeg is competitive with and often surpasses the specialist SAM 3 model on identical items. It reaches 71.5 cIoU on ReasonSeg val without additional segmentation-specific training. Backend model updates can improve accuracy without retraining the pipeline. The code is available at https://abdullahamdi.com/llmseg .
Figures & tables
Figure 1: Text-prompted segmentation from a frozen MLLM. LLMSeg segments an image from a free-form text query alone, with a frozen multimodal LLM as its only vision component: no segmentation head, no mask decoder, no training. Top: four results, each under its query. Bottom: the model emits geometry rather than pixels; deterministic code rasterizes it, and the rendered overlay is fed back so the model can inspect and correct its own boundary.
Figure 2: LLMSeg pipeline overview. Detection localizes instances, segmentation traces box-confined B-splines, and refinement corrects the mask within signed focus boxes ( adds pixels, removes them). Each MLLM call serves all instances (one is shown).
Figure 3: Reasoning queries work off the shelf. Two ReasonSeg items segmented by LLMSeg with Gemini-3.7-Flash. The same model reads the query, reasons about the image and outputs mask geometry, with no agent wrapper or query rewriting.
| ReasonSeg | RefCOCO | RefCOCO+ | RefCOCOg | ||||||||
| Method | MLLM/Backend | val | test | val | testA | testB | val | testA | testB | val (U) | test (U) |
| Seg-Zero | Qwen2.5-VL 7B | 62.0 | 52.0 | – | – | – | – | – | – | – | – |
| LISA | LLaVA 7B | – | – | 74.9 | 79.1 | 72.3 | 65.1 | 70.8 | 58.1 | 67.9 | 70.6 |
| LISA | LLaVA1.5 13B | 72.9 | 62.2 | – | – | – | – | – | – | – | – |
| GLaMM | Vicuna 7B | 47.2 | – | 79.5 | 83.2 | 76.9 | 72.6 | 78.7 | 64.6 | 74.2 | 74.9 |
| PSALM | Phi1.5 1.3B | – | – | 83.6 | 84.7 | 81.6 | 72.9 | 75.5 | 70.1 | 73.8 | 74.4 |
Table 1: Reasoning and referring expression segmentation. Results on ReasonSeg ( Lai et al., 2024 ) and RefCOCO/+/g ( Kazemzadeh et al., 2014 ; Mao et al., 2016 ) . All values are cIoU. The baselines are Seg-Zero ( Liu et al., 2025 ) , LISA ( Lai et al., 2024 ) , GLaMM ( Rasheed et al., 2024 ) , PSALM ( Zhang et al., 2024c ) , RICE ( Xie et al., 2025 ) , HyperSeg ( Wei et al., 2024 ) , X-SAM ( Wang et al., 2025a ) , Grounded SAM ( Ren et al., 2024 ) , Gemini Seg ( Voigtlaender et al., 2025 ) , Kang et al. ( Kang et al., 2025 ) , RSVP ( Lu et al., 2025 ) , and both SAM 3 and SAM 3 Agent ( Carion et al., 2026 ) . The top block contains methods fine-tuned for referring or reasoning segmentation. Unmarked rows are published full-split numbers quoted from SAM 3 Carion et al. (2026) . ∗ Our baseline evaluation uses the complete ReasonSeg splits. On RefCOCO/+/g, it uses the same 500-item random subset per split as LLMSeg. Bold is the best and underline the second best per column.
Figure 4: LLMSeg Qualitative comparisons. We show compariosns of LLMSeg with baselines on various setups. Top : input with ground truth in amber . Middle : the baselines SAM 3 in the first four columns and LLMSeg (w/o refine) in the last two. Bottom : LLMSeg.
| Semantic segmentation | Zero-shot detection | ||||||||||
| ADE20K-150 | Cityscapes | ODinW13 | RF-100VL | ||||||||
| Method | gIoU | cIoU | mIoU | AP 0 | gIoU | cIoU | pmF1 | AP 0 | gIoU | cIoU | pmF1 |
| APE-D (LVIS-trained) | – | – | 44.2 | – | – | – | – | – | – | – | – |
| Gemini 2.5 Pro | – | – | – | 33.7 | – | – | – | 11.6 | – | – | – |
| gDino-T | – | – | – | 49.7 | – | – | – | 15.7 | – | – | – |
| gDino1.5-Pro | – | – | – | 58.7 | – | – | – | – | – | – | – |
Table 2: Semantic segmentation and zero-shot detection. The baselines are APE-D ( Shen et al., 2024 ) , Gemini 2.5 Pro ( Comanici et al., 2025 ) , and gDino-T ( Liu et al., 2024a ) . LLMSeg uses Gemini-3.7-Flash. Unmarked baselines are published full-split numbers. ∗ Our evaluation uses complete splits, except on ADE20K, where we use the same 1,000 random items as the shaded rows. ADE20K reports oracle-class union-IoU and Cityscapes reports 19-class mIoU. Detection reports mean per-subset box AP 0 and pooled mask metrics. Bold is the best and underline the second best per column.
| LVIS | COCO | |||||
| Method | cgF1 | box AP | mask AP | cgF1 | box AP | mask AP |
| OWLv2 | 20.1 | 35.2 | – | – | 38.2 | – |
| OWLv2 † | 29.3 | 45.5 | 43.4 | – | 46.1 | – |
| gDino-T | 14.7 | 20.5 | – | – | 45.7 | – |
| LLMDet-L | 35.1 | 42.0 | 36.3 | – | 55.6 | – |
| APE-D † | – | 59.6 | 53.0 | – | 58.3 | – |
Table 3: Open-vocabulary instance segmentation. The baselines are OWLv2 ( Minderer et al., 2023 ) , gDino-T ( Liu et al., 2024a ) , LLMDet-L ( Fu et al., 2025 ) , APE-D ( Shen et al., 2024 ) , DINO-X ( Ren et al., 2025 ) , and Gemini 2.5 Flash ( Comanici et al., 2025 ) . ∗ Evaluated on a matched subset. Unmarked rows are published results from SAM 3 ( Carion et al., 2026 ) . † LVIS-trained. cgF1 is presence-gated F1.
Figure 5: LLMSeg quality is the backend’s quality. Stage-2 gIoU on all 200 ReasonSeg val items. We show 18 of the 21 tested backends for readability. Whiskers show 95% item-bootstrap intervals. Dashed lines mark SAM 3 and SAM 3 Agent (Gemini-3.7-Flash). Recent MLLMs like GPT-6 Astra surpasses SAM 3 Agent even without refinement.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: A step-by-step view of LLMSeg on one ReasonSeg test item , “the spare wheel” , with the Gemini-3.7-Flash backend. Top: stage 1’s box, then stage 2’s control points and the resulting closed cubic B-spline. Middle: three of the four passes, each of which proposes signed focus boxes over the current mask and composes the re-traced loops. The net effect is shown in green (added) and red (removed). Bottom: one loop enlarged, the converged mask, the ground truth, and the IoU after each pass.
| Benchmark (splits) | Query (words) | Split | Neg. | Scored | Ground truth | Metric |
| ReasonSeg val / test | reasoning sentence (18.3 / 20.7) | 200 / 779 | 2.0 / 0.3 | all | one mask per query | gIoU, cIoU |
| RefCOCO val / testA / testB | referring expression (3.6) | 10,834 / 5,657 / 5,095 | 0 | 500 each | one instance | gIoU, cIoU |
| RefCOCO+ val / testA / testB | no location words (3.6) | 10,758 / 5,726 / 4,889 | 0 | 500 each | one instance | gIoU, cIoU |
| RefCOCOg val / test | written description (8.5) | 4,896 / 9,602 | 0 | 500 each | one instance | gIoU, cIoU |
| LVIS val | category name (1.4) | 129,547 | 45.9 | 1,500 | every instance | cgF1, pmF1, AP |
| COCO val2017 | category name (1.2) | 29,631 | 50.6 | 1,500 | every instance | cgF1, pmF1, AP |
Table 5: Benchmarks. Query type with mean query length in words, the number of items in each split after loading, the share of items whose correct answer is an empty mask, the number of items we score, and the reported metric. An item is one (image, query) pair. Negative shares are percentages of the total number of items in the split.
| Training | val | test | |||||
| Method | MLLM | RES | ReasonSeg | gIoU | cIoU | gIoU | cIoU |
| Published, full splits ( Carion et al., 2026 ) | |||||||
| SEEM ( Zou et al., 2023 ) | – | ✗ | ✗ | 25.5 | 21.2 | 24.3 | 18.7 |
| Grounded SAM ( Ren et al., 2024 ) | – | ✗ | ✗ | 26.0 | 14.5 | 21.3 | 16.4 |
| OVSeg ( Liang et al., 2023 ) | – | ✗ | ✗ | 28.5 | 18.6 | 26.1 | 20.8 |
| GLaMM ( Rasheed et al., 2024 ) | Vicuna 7B | ✓ | ✗ | 47.4 | 47.2 | – | – |
Table 6: Reasoning segmentation on ReasonSeg ( Lai et al., 2024 ) . Top block: published numbers quoted from Carion et al. (2026) on the full splits, reproducing its complete baseline list. Bottom block (below the rule): our own evaluations (200 validation items and 779 test items). Both blocks cover the full splits. SAM 3 and SAM 3 Agent are both baselines from Carion et al. (2026) . Only the shaded rows are ours. The two training columns follow Carion et al. (2026) : RES = fine-tuned for referring expression segmentation, ReasonSeg = fine-tuned on the ReasonSeg training split. A method is zero-shot when both entries are ✗ . LLMSeg (w/o refine) is our pipeline stopped after the segment stage. The best zero-shot result in each column is shown in bold , separately for each block.
| RefCOCO | RefCOCO+ | RefCOCOg | ||||||||
| Method | MLLM / backbone | RES | val | testA | testB | val | testA | testB | val (U) | test (U) |
| Published, full splits, cIoU ( Carion et al., 2026 ) | ||||||||||
| LISA ( Lai et al., 2024 ) | LLaVA 7B | ✓ | 74.9 | 79.1 | 72.3 | 65.1 | 70.8 | 58.1 | 67.9 | 70.6 |
| GSVA ( Xia et al., 2024 ) | 13B | ✓ | 79.2 | 81.7 | 77.1 | 70.3 | 73.8 | 63.6 | 75.7 | 77.0 |
| GLaMM ( Rasheed et al., 2024 ) | Vicuna 7B | ✓ | 79.5 | 83.2 | 76.9 | 72.6 | 78.7 | 64.6 | 74.2 | 74.9 |
| SAM4MLLM ( Chen et al., 2024 ) | LLaVA1.6 7B | ✓ | 79.6 | 82.8 | 76.1 | 73.5 | 77.8 | 65.8 | 74.5 | 75.6 |
Table 7: Referring expression segmentation on RefCOCO/+/g ( Kazemzadeh et al., 2014 ; Mao et al., 2016 ) . The top block reports published full-split cIoU results quoted from Carion et al. (2026) , separated into methods fine-tuned on the RES task (RES ✓ ) and zero-shot methods (RES ✗ ). Note that nearly all of the MLLM-based methods were trained on the RefCOCO box annotations. The bottom blocks report our runs on matched 500-item random subsets per split (shuffled, seed 0), with gIoU in the upper block and cIoU in the lower block. All methods in the bottom blocks are zero-shot for RES and use the same selected items. SAM 3 and SAM 3 Agent are baselines from Carion et al. (2026) , reproduced by us on these items. Only the shaded rows show our methods. Best results within each block are in bold . The best zero-shot result in each published-block column is underlined . RefCOCOg val (U) and test (U) are the splits used in our evaluation.
| LVIS | COCO | SA-Co/Gold | |||||||||
| Method | MLLM | cgF1 | pmF1 | box AP | mask AP | cgF1 | pmF1 | box AP | mask AP | cgF1 | pmF1 |
| Human | – | – | – | – | – | – | – | – | – | 72.8 | – |
| OWLv2 ( Minderer et al., 2023 ) | – | 20.1 | – | 35.2 | – | – | – | 38.2 | – | 17.3 | – |
| OWLv2 (LVIS-trained) | – | 29.3 | – | 45.5 | 43.4 | – | – | 46.1 | – | 24.6 | – |
| gDino-T ( Liu et al., 2024a ) | – | 14.7 | – | 20.5 | – | – | – | 45.7 | – | 3.3 | – |
| LLMDet-L ( Fu et al., 2025 ) | – | 35.1 | – | 42.0 | 36.3 | – | – | 55.6 | – | 6.5 | – |
Table 8: Open-vocabulary instance segmentation. Rows without asterisks or shading report published full-split results quoted from Carion et al. (2026) . Rows marked with ∗ show our evaluations of the SAM 3 baselines on matched random subsets with hard negatives, using the same items as our methods (shaded rows). cgF1 is presence-gated F1, and pmF1 is its positive-only counterpart. Within the starred and shaded rows (our matched block), the best and second-best result per column are shown in bold and underlined , respectively. Published rows and the human reference are not ranked.
Figure 8: LLMSeg across benchmarks and task formats . The same Gemini-3.7-Flash backend and settings are used throughout. The prompt appears at left. Ground truth is shaded amber, and predictions are shown in green with a red outline. Each chip reports the panel’s IoU. Column 3 shows the stage-2 mask, and column 4 shows the final mask. Together, they isolate the effect of refinement.
Figure 9: Dense semantic labeling on Cityscapes ( Cordts et al., 2016 ) , Gemini-3.7-Flash backend, official 19-class palette. LLMSeg labels every class in an image from a single query and resolves overlaps by deterministic mask compositing. Each panel is therefore a full label map, rather than a single mask, and its chip reports the image’s mIoU. The losses occur on amorphous ground classes, which lack clear visual boundaries.
Figure 10: Where LLMSeg separates from both SAM 3 baselines. Selected items from ReasonSeg, RefCOCO/+/g, SA-Co/Gold, and ADE20K. SAM 3 ( Carion et al., 2026 ) receives the same text query as LLMSeg, and each chip reports that system’s recorded IoU.
Figure 11: Images containing multiple objects of the queried class. LVIS, COCO, and SA-Co/Gold examples whose ground truth contains at least three objects. Each label gives the benchmark and number of ground-truth objects. Each chip reports the corresponding system’s recorded union IoU, which does not separate touching objects that a system merges into one region.
Figure 12: Hard negatives: the prompted concept is absent. Six SA-Co/Gold examples whose phrases have no matching annotation in the image. Because the ground truth is empty, no ground-truth panel is shown. Each chip indicates only whether the system produced a mask. Each scene contains the broader category , but not the qualified concept.
Figure 13: Rare domains segmentation on RF-100VL subsets. The rows span documents, aerial imagery, medical imaging, and industrial inspection, with one subset per row. Each prompt uses that subset’s category name verbatim, and each ground-truth panel shows the annotated detection box filled in. Some category names are domain jargon, so a failure here may reflect vocabulary rather than localization.
Figure 14: What one refinement pass does. Four items, one per task format, for which the final mask substantially outperforms the stage-2 mask. Each is shown at stage 2, after the first two passes, and after the final pass. Each chip identifies the state and reports its IoU. The intermediate masks are replayed from each item’s refinement record using the pipeline’s own composition code.
Figure 15: The refinement loop stops on its own. ReasonSeg val, all 200 items, Gemini-3.7-Flash. (a) Share of items still inside the loop at each pass. A run stops once the mask changes by less than of its box area for three consecutive passes. (b) The same run’s mask after the first passes, scored at every budget, with the mean MLLM calls on the top axis.
Figure 16: Per-item refinement residuals on the development set. Each curve is one development image, run with Gemini-3.7-Flash at budget . The curve is the -pass moving average of the mask residuals on a logarithmic axis, and the dashed line is the threshold . Blue curves are images that stop on their own with stopping patience , orange curves are images that exhaust the budget, and marks the first smoothed pass below . The residuals decay as the masks stabilize, which is the signal the stopping criterion uses in LLMSeg.
Figure 17: Backend progress is LLMSeg progress. LLMSeg final segmentation gIoU on ReasonSeg val against backend release date, one coloured line per product line with two or more releases. All other backends are gray. Nothing in the pipeline changes between the points of a line.
Figure 18: Complete backend comparison in LLMSeg. All 21 backends we ran, including Gemini 3.8 Flash, Gemma 4 26B, and Qwen3.8 27B. Stage-2 gIoU, 95% item-bootstrap intervals, and baselines use the same 200 ReasonSeg val items.
| Backend | Developer | Release | Stage-2 gIoU | Final gIoU | |
| GPT-6 Astra † | OpenAI | 2026-09 | 74.2 [70.4, 77.9] | 72.9 [69.2, 76.6] | -1.2 |
| Gemini 3.7 Flash | 2026-08-13 | 65.7 [62.1, 69.5] | 69.4 [65.9, 72.8] | +3.7 | |
| Gemini 3.8 Flash † | 2026-09 | 63.4 [59.4, 67.3] | 66.7 [63.0, 70.4] | +3.3 | |
| Gemini 3.5 Flash | 2026-05 | 62.9 [59.1, 66.6] | 66.2 [62.4, 70.0] | +3.3 | |
| GPT-5.6 Sol † | OpenAI | 2026-07 | 61.2 [57.3, 64.9] | 61.6 [57.7, 65.5] | +0.3 |
| Gemini 3 Flash | 2025-12 | 58.9 [55.1, 62.7] | 58.6 [55.1, 61.9] | -0.3 |
Table 9: Twenty-one backends. LLMSeg on all 200 ReasonSeg val items, using the same pipeline and settings. Only the MLLM differs. Both masks are shown, with 95% bootstrap intervals and the signed effect of refinement. marks a longer per-call budget.
Figure 19: Which kind of capability predicts LLMSeg quality? Final gIoU on ReasonSeg val is plotted against four public-benchmark capability axes, with dashed OLS fits. Slopes (gIoU per unit ) are 7.7 for vision and 6.9 for reasoning , followed by 6.0 for software and 0.6 for agentic skill. Reasoning and vision track segmentation quality most closely. Agentic skill shows almost no association.
Figure 20: Segmentation quality is not bought. Backend input price is plotted against LLMSeg final gIoU on ReasonSeg val. The dashed staircase marks the cheapest-so-far frontier, whose backends are shown in blue. Price does not order quality. Gemini 3.7 Flash is the cheapest backend that reaches final gIoU, which motivates our LLMSeg choice of backend for the scalable main results.
Figure 21: What the MLLM backend gap looks like in LLMSeg. The same pipeline is used throughout, with one column per MLLM backend, ordered by split-level stage-2 gIoU, best first. Every panel shows the stage-2 mask, so the comparison is of what each backend traces rather than how it refines. Grounding is largely shared, but mask outlines deteriorate down the rows with week MLLMs.
| Variant | gIoU | cIoU | calls/image | s/image |
| LLMSeg (reported configuration) | 70.7 0.74 | 74.7 1.62 | 14.6 0.32 | 177 9 |
| w/o deterministic postprocessing | 70.2 0.27 | 74.1 2.20 | 14.8 0.37 | 162 12 |
| + box-adjust pre-pass | 70.0 0.27 | 70.2 0.38 | 15.7 0.34 | 188 17 |
| w/o postprocessing + box-adjust | 69.5 0.11 | 73.2 1.64 | 15.9 0.10 | 169 12 |
| + topology conditioning | 69.2 1.34 | 72.6 0.23 | 14.3 0.59 | 175 17 |
| w/o refinement ( ) | 63.8 0.68 | 67.2 2.72 | 2.0 0.00 | 24 2 |
Table 10: Ablations on ReasonSeg val (all 200 items, Gemini-3.7-Flash backend, ). The shaded row is the configuration reported in the paper (topology conditioning off). Cells are three-run means and sample deviations. s/image is backend-call wall-clock time per image.
| Variant | gIoU | cIoU | calls/image |
| LLMSeg (reported configuration) | 79.3 | 79.8 | 17.5 |
| w/o deterministic postprocessing | 76.6 | 77.9 | 17.1 |
| + box-adjust pre-pass | 79.2 | 79.7 | 18.4 |
| w/o postprocessing + box-adjust | 77.9 | 78.3 | 19.1 |
| + topology conditioning | 76.7 | 77.3 | 16.8 |
| w/o refinement ( ) | 66.9 | 64.6 | 2.0 |
Table 11: Ablations on RefCOCO val (all 200 items, Gemini-3.7-Flash backend, , one run per configuration). Rows and blocks as in the ReasonSeg ablation: the shaded row is the reported configuration.
Figure 22: Polygons vs. B-splines in LLMSeg. Stage-2 traces from the polygon-loop ablation and the B-spline run on identical items with the same backend. Each chip shows that run’s recorded stage-2 IoU. Cubic B-splines cover curved boundaries better than polygons with fewer control points.
Figure 23: Explicitly carved holes. The query is “the sewage outlet” . The cyan-box region is enlarged. SAM 3 fills the grate solidly, whereas LLMSeg carves one hole per slot.
Figure 24: LLMSeg localizes like SAM 3 and loses the boundary. COCO-style AP at each IoU threshold on AP-eligible images in the matched LVIS and COCO subsets, with boxes taken from the predicted masks. LLMSeg is the reported configuration. Solid lines are box AP and dashed lines mask AP. The box curves run together while the mask curves fan apart, indicating good localization but loose boundary for LLMSeg errors.
| measured / estimated cost per image | ||||
| Method | calls | median | est. $ | final |
| MLLM / SAM 3 | s | / 1k img | gIoU | |
| LLMSeg (GPT-6 Astra) | 11.7 / 0 | 218 | 323.9 | 72.9 |
| LLMSeg (Gemini 3.7 Flash) | 15.0 / 0 | 153 | 15.6 | 69.4 |
| LLMSeg (Gemini 3.8 Flash) | 16.2 / 0 | 419 | 33.7 | 66.7 |
| LLMSeg (Gemini 3.5 Flash) | 17.6 / 0 | 246 | 77.9 | 66.2 |
Table 12: What a mask costs. ReasonSeg validation set (200 items), covering all 21 LLMSeg backends and the SAM 3 and SAM 3 Agent baselines. Call counts and median latency are measured per image. Estimated spend is per 1,000 images, using one token budget shared across all rows and a fixed serving price per SAM 3 forward pass. LLMSeg rows are shaded.
Figure 25: Quality against what it costs. ReasonSeg val, 200 items, every LLMSeg backend and baseline, plotted against wall-clock time ( top ) and estimated serving cost ( bottom ). Both axes are logarithmic and the dashed staircase is the cheapest-so-far frontier over the LLMSeg backends. Circles are LLMSeg backends, stars the SAM 3 and SAM 3 Agent baselines, which the frontier excludes.
Figure 26: Failure modes. Top left pair: thin structures (retinal vessels) defeat spline outlines even after refinement. Top right pair: on Cityscapes, amorphous and heavily-occluded background classes favour pixel decoders over compact parametric loops. Bottom row: the query is grounded correctly, but a closed loop cannot cut into the deep, narrow concavities of an articulated hands.
More in this topic
Image Segmentation
Image segmentation assigns pixels or voxels to objects and regions, supporting measurement, scene understanding, and decisions across image domains.
See all 399 papers