Ensuring trustworthiness and versatility in Open-World Visual Recognition is a core goal of Multimodal Large Language Models (MLLMs). While integrating segmentation is desirable, prevailing methods require invasive fine-tuning, which often leads to catastrophic forgetting of the model's general-purpose capabilities. This trade-off between task-specific performance and general reliability fundamentally undermines the goal of building unified, trustworthy systems. We introduce LENS (Leveraging kEypoiNts for MLLMs' Segmentation), a novel plug-and-play paradigm designed to equip MLLMs with pixel-level ability without compromising their foundational integrity. LENS attaches a lightweight head to a completely frozen MLLM, interpreting the intrinsic spatial cues embedded in attention maps. It extracts keypoints and translates them into prompt-compatible features for a mask decoder. Extensive experiments validate that LENS achieves segmentation performance competitive with retraining-based methods. Crucially, it largely preserves the MLLM's open-world generalization, avoiding the degradation observed in fine-tuned counterparts. By decoupling segmentation from generation, LENS offers a robust and efficient path toward truly multi-talented and reliable vision models.
Figures & tables
Figure 1: Conventional architecture for MLLM segmentation vs. LENS . Conventional methods ( e.g . LISA ( Lai et al, 2024 ) ) fine-tune an MLLM for both generation and segmentation tasks, leading to conflicting objectives that undermine the model’s general capabilities and training stability. LENS decouples these roles: a frozen MLLM is dedicated to reasoning, while a lightweight head is trained exclusively for segmentation. The head can be adaptively invoked by the model when needed, allowing the MLLM to serve as a unified vision model capable of handling diverse tasks
Symbol
Description
I,T
Input image and instruction text
Fi,Ft
Intermediate MLLM features for image and text
Hin1
Input features to the head (concatenation of Fi and Ft )
Ac1
Text-to-image attention from the start-of-answer token (grounding map)
Hout1,Hout2
Output features of Head Layer 1 and Layer 2
fs2
Feature of the start-of-answer token (used as global query)
Table 1: Notations. Summary of key symbols used in the LENS architecture ( c.f . Fig. 2 )
Figure 2: LENS architecture. (a) Overall architecture built on LLaVA-1.5-7B ( Liu et al, 2024a ) , consisting of three stages: ① a segmentation head that refines attention and semantic features, ② keypoint extraction and description, and ③ a mask decoder that takes fused keypoint descriptions and coordinates to predict the final mask. Trainable and frozen modules are indicated, and the MLLM runs only in inference mode. (b) Zoom-in view of the attention and keypoint components
Figure 3: Tracing attention back to image patches across different MLLM backbones. The attention map Ac1 is computed identically across backbones, while its spatial reshaping follows the corresponding visual tokenization strategy: a fixed 24×24 grid for LLaVA-1.5, a variable-size grid for Qwen2-VL, and an AnyRes crop arrangement for LLaVA-OV. The recovered 2D attention map is then processed by the same keypoint extraction pipeline
Figure 4: Detailed architecture of the Descriptor Model. Illustrating the interaction between the global semantic query ( fs2 ) and local neighborhood features ( dN(i) ). The cross-attention mechanism serves as a verifier to determine if a keypoint corresponds to the target object, filtering out background noise effectively
Figure 5: Showcases of LENS . Attention maps align well with ground-truth regions, where white points mark keypoints inside and red points mark those outside. These results illustrate how LENS links semantics with segmentation
Method
Backbone
Training Mem (GB) ↓
Training Data
Seg ↑
MME
MMBench
MMMU
MMStar
Seg
FP-Seg
VQA
random guess
–
–
–
–
–
–
1050.0
25.0
26.8
24.6
LLaVA-1.5-7B
–
–
✓
–
1808.4
66.5
35.7
33.1
Qwen2-VL-2B
–
–
✓
–
1872.0
74.9
41.1
48.0
LLaVA-OV-7B
–
–
✓
–
1993.6
80.9
47.9
61.9
SESAME
LLaVA-1.5-7B
30×8
✓
✓
30.4
1394.4
28.3
11.2
20.3
Table 2: Comprehensive comparison. LENS achieves competitive segmentation performance under a substantially lighter training setting, while preserving the backbone’s general abilities. Training memory values with underline mark inference overhead of the MLLM. Seg denotes segmentation data (semantic, referring, reasoning), FP-Seg augments it with false-premise samples, and VQA represents corpus from generative vision–language tasks. READ uses the largest training set, whereas LENS relies solely on Seg . The bottom block reports LENS on two additional backbones, Qwen2-VL-2B ( Wang et al, 2024b ) and LLaVA-OV-7B ( Li et al, 2024 ) , with the corresponding untouched base models as references ( c.f . § 4.3 )
Method
val
test
overall
short query
long query
overall
gIoU
cIoU
gIoU
cIoU
gIoU
cIoU
gIoU
cIoU
SESAME ( Wu et al, 2024b )
34.8
39.1
28.3
27.6
31.6
32.7
30.5
30.4
LLaVA + OVSeg ( Liang et al, 2023 )
38.2
23.5
24.2
18.7
44.6
37.1
39.7
31.8
LISA-7B ( Lai et al, 2024 )
52.9
54.0
40.6
40.6
49.4
51.0
47.3
48.4
LISA-LLaVA-1.5-7B ( Lai et al, 2024 )
61.3
62.9
48.3
46.3
57.9
59.7
55.6
56.9
Table 3: Comparisons on the ReasonSeg dataset. The best performance is highlighted in bold , and the second best is underlined . The bottom block reports LENS with two additional MLLM backbones (paired with SAM ViT-H).
Method
RefCOCO
RefCOCO+
RefCOCOg
Mean
val
testA
testB
val
testA
testB
val(U)
test(U)
MCN ( Luo et al, 2020 )
62.4
64.2
59.7
50.6
55.0
44.7
49.2
49.4
54.4
VLT ( Ding et al, 2021 )
67.5
70.5
65.2
56.3
61.0
50.1
55.0
57.7
60.4
CRIS ( Wang et al, 2022 )
70.5
73.2
66.1
62.3
68.1
53.7
59.9
60.4
64.3
LAVT ( Yang et al, 2022 )
72.7
75.8
68.8
62.1
68.4
55.1
61.2
62.1
66.5
ReLA ( Liu et al, 2023a )
73.8
76.5
70.2
66.0
71.0
57.7
65.0
66.0
68.3
Table 4: Comparison of SOTA referring segmentation (cIoU) on RefCOCO(+/g).
Layer Index
6
10
12
14 (Ours)
18
30
cIoU
42.1
55.4
57.1
57.3
51.8
40.0
Table 5: Impact of layer selection on ReasonSeg test set
Num. Keypoints ( m )
4
8
16 (Ours)
24
32
cIoU
55.6
57.0
57.3
51.1
52.5
Table 6: Sensitivity to the number of keypoints ( m )
Figure 6: Representative samples from our routing benchmark. Each column shows one of the four difficulty tiers (non-segmentation dialogue, direct segmentation, reasoning segmentation, and mixed-intent), illustrating how the frozen MLLM router decides whether a query should be answered by dialogue or trigger the segmentation branch
Figure 7: Qualitative results on the ReasonSeg dataset. Visualizing the progression from input instruction to attention maps, keypoint extraction, and final segmentation
Figure 8: Failure cases. The primary limitation of LENS is its dependency on the frozen backbone
Method
Peak Mem. (GB, 8 GPUs)
Train. Params (M)
Train. Samples
Train. Time (GPU-h)
Latency (ms) ↓
SESAME
30×8
∼ 305 (LoRA, in MLLM)
250k
∼ 30
1340
LISA
30×8
∼ 305 (LoRA, in MLLM)
800k
∼ 53
1104
READ
30×8
∼ 305 (LoRA, in MLLM)
800k
∼ 56
1631
LENS (Ours)
10×8+16
668 (external head)
256k
∼ 16
182
Table 7: Training cost and inference efficiency. All methods build on the LLaVA-1.5-7B architecture (READ is initialized from the SESAME checkpoint) and use the same SAM encoder under their official training configurations ( 8× A800, bf16). Peak Mem. : total peak training memory across 8 GPUs. Train. Params : trainable parameters and where they reside. Train. Samples : total training samples processed until convergence under each method’s official recipe (epochs × steps × effective batch size; for LENS , 10×100×(8×32)=256 k). Latency : per-image inference latency with batch size 1, bf16, on one A800
Method
Mask quality ↑
Keypoint interp. ↑
GT (reference)
4.9
–
LISA
3.7
–
LENS (Ours)
3.9
4.1
Table 8: Human evaluation on 100 segmentation cases (ReasonSeg/RefCOCO; N=3 raters; 5-point Likert means ↑ ). Keypoint interpretability, which we use to gauge agent usability, applies only to methods exposing intermediate keypoints
Figure 9: Training-loss curves of LENS (LLaVA-OV-7B backbone). The total training loss and its two components, the attention loss Lattn and the segmentation loss Lseg , decrease smoothly and plateau over the 10 training epochs, indicating stable optimization without divergence or oscillation. Light curves show raw logged values; solid curves apply a centered moving average
Backbone
Setting
gIoU
LLaVA-1.5-7B
full image (single pass)
40.8
+ coarse-to-fine
43.9 (+3.1)
LLaVA-OV-7B
full image (single pass)
37.0
+ coarse-to-fine
44.3 (+7.3)
Table 9: Training-free coarse-to-fine mitigation for small objects. Per-instance gIoU on the ReasonSeg small-object subset (short queries with target-mask area <10% ; 140 cases). The same frozen pipeline is evaluated on the full image and with coarse-to-fine refinement; no additional parameters or fine-tuning are introduced