Xiaomi-Lens-0 Technical Report

Pei Fu; Yike Liu; Yue Sun; Hanmo Wang; Xiuwen Xi; Ruoceng Zhang; Shaojie Zhang; Chao Jiang

cs.CVOct 16, 2026

Xiaomi-Lens-0 Technical Report

Authors: Pei Fu, Yike Liu, Yue Sun, Hanmo Wang, Xiuwen Xi, Ruoceng Zhang, Shaojie Zhang, Chao Jiang,

Organizations: Xiaoai Plus, SeerRay Team

Abstract

Multimodal large language models (MLLMs) answer questions about whole images well, yet they still struggle with fine-grained region understanding. A key reason is that text-only supervision never specifies which regions support an answer. To close this gap, we present Xiaomi-Lens-0, a 4B MLLM built on Qwen3.5 that predicts masks from its own visual representations. Specifically, a lightweight decoder combines vision-encoder patch features with visual latents, the language model's mask-token states, so mask supervision reaches the shared backbone without an additional segmentation encoder. Because each latent decodes to a mask, the model can interleave reasoning with region-specific latents and thus think with visual latents, while grounded reasoning traces supervise the regions it visits during chain-of-thought. As a result, across all eight RefCOCO, RefCOCO+, and RefCOCOg splits, Xiaomi-Lens-0 gains 5.6 points on average over Qwen3.5 in referring comprehension and achieves the highest segmentation cIoU among compared models. Moreover, it surpasses Qwen3.5 across all perception benchmarks. Homepage: this https URL

Figures & tables

Figure 2: Xiaomi-Lens-0 architecture. The language model consumes spatially merged visual tokens. Each generated <|mask|> token has a contextual state that queries the encoder’s patch-level features through a lightweight mask decoder. Patch-wise compatibility scores are reassembled into a spatial grid and upsampled with two transposed-convolution layers.

Figure 3: Training data for Stage 1 mask decoder pretraining (left) and Stage 2 supervised fine-tuning (right). Donut charts show the proportions of public and in-house data; the adjacent lists give the public datasets and their sample counts.

Figure 4: RES data pipeline. (a) Candidate objects are localized, their descriptions are checked against image crops, and masks are generated. (b) The masked view and original image guide referring-instruction synthesis. (c) Re-grounding and contextual verification check that each instruction identifies its intended target.

Figure 5: VQA data pipeline. (a) After image and region screening, an MLLM generates question–answer pairs with multiple-choice or numeric answers, optionally referring to regions by temporary IDs. (b) Rule-based filters remove flawed questions, options are shuffled, and an independent checker must reproduce the reference answer. (c) A reasoning model generates traces with box references. Traces that are complete, use valid boxes, and match the verified answer are paired with masks whose bounding boxes agree with the referenced boxes.

Figure 6: Thinking with visual latents. The top example identifies the position of food relative to a woman. The middle example localizes people in two boats and sums the per-boat counts. The bottom example localizes and counts individual picture frames. Colored boxes and masks show the regions referenced in each reasoning trace.

Figure 7: LensBench design and construction. (a) Examples from four domains combine single and multiple targets with explicit and implicit instructions. Each unit contains two ground-truth masks and four instructions. (b) Visual grounding, instance curation, and grouping establish the targets before instruction synthesis. Automated checks of masks, instructions, and re-grounded instances inform human review of each complete unit.

Methods	RefCOCO			RefCOCO+			RefCOCOg
Methods	Val	TestA	TestB	Val	TestA	TestB	Val	Test
Grounding-DINO-L [ 25 ]	90.6	93.2	88.2	82.8	89.0	75.9	86.1	87.0
Florence-2 [ 61 ]	93.4	95.3	92.0	88.3	92.9	83.6	91.2	91.7
LISA ${}_{\text{LLaVA-7B}}$ [ 18 ]	85.4	88.8	82.6	74.2	79.5	68.4	79.3	80.4
GSVA ${}_{\text{Vicuna-7B}}$ [ 60 ]	86.3	89.2	83.8	72.8	78.8	68.0	81.6	81.8
PixelLM ${}_{\text{LLaVA-7B}}$ [ 41 ]	89.8	92.2	86.4	83.2	87.0	78.9	84.6	86.0

Table 1: REC results on RefCOCO and RefCOCO+ [ 67 ] , and RefCOCOg [ 30 , 33 ] . Scores are [email protected] (%, higher is better). Best scores are bold; second-best scores are underlined.

Methods	RefCOCO			RefCOCO+			RefCOCOg
Methods	[email protected]	[email protected]	F1@mIoU	[email protected]	[email protected]	F1@mIoU	[email protected]	[email protected]	F1@mIoU
GroundingDINO [ 25 ]	51.2	23.3	44.3	49.2	20.9	41.4	58.0	23.2	49.7
Rex-Omni [ 15 ]	84.4	32.9	71.4	76.9	29.5	64.1	84.7	34.7	72.3
LocateAnything Fast [ 51 ]	92.5	44.0	80.6	84.4	39.5	73.1	88.5	40.7	76.2
LocateAnything Hybrid [ 51 ]	92.7	44.3	81.4	85.8	40.6	75.1	89.4	41.5	77.7
LocateAnything Slow NTP [ 51 ]	92.4	38.4	80.2	85.2	35.7	73.9	88.7	35.6	76.1

Table 2: REC results across IoU thresholds on RefCOCO and RefCOCO+ [ 67 ] , and RefCOCOg [ 30 , 33 ] . [email protected] and [email protected] use the indicated IoU thresholds; F1@mIoU is the mean F1 over thresholds from 0.50 to 0.95 in steps of 0.05. All scores are percentages, and the higher is better. Best scores are bold; second-best scores are underlined.

Methods	RefCOCO			RefCOCO+			RefCOCOg
Methods	Val	TestA	TestB	Val	TestA	TestB	Val	Test
LISA ${}_{\text{LLaVA-7B}}$ [ 18 ]	74.9	79.1	72.3	65.1	70.8	58.1	67.9	70.6
GSVA ${}_{\text{Vicuna-7B}}$ [ 60 ]	77.2	78.9	73.5	65.9	69.6	59.8	72.7	73.3
PixelLM ${}_{\text{LLaVA-7B}}$ [ 41 ]	73.0	76.5	68.2	66.3	71.7	58.3	69.3	70.5
VisionLLM v2 ${}_{\text{Vicuna-7B}}$ [ 58 ]	76.6	79.3	74.3	64.5	69.8	61.5	70.7	71.2
Text4Seg ${}_{\text{InternVL2-8B}}$ [ 20 ]	79.2	81.7	75.6	72.8	77.9	66.5	74.0	75.3

Table 3: RES results on RefCOCO and RefCOCO+ [ 67 ] , and RefCOCOg [ 30 , 33 ] . Scores are cIoU (%, higher is better). Best scores are bold; second-best scores are underlined.

Methods	IoU 0.50			IoU 0.95			mIoU
Methods	Recall	Precision	F1	Recall	Precision	F1	Recall	Precision	F1
GroundingDINO [ 25 ]	79.6	83.2	81.4	24.7	28.1	26.3	61.1	63.7	60.6
Rex-Omni [ 15 ]	69.9	81.0	75.0	17.9	21.7	19.7	53.9	59.4	56.3
LocateAnything Fast [ 51 ]	66.2	73.3	69.6	24.7	26.9	25.8	50.9	59.3	53.1
LocateAnything Hybrid [ 51 ]	74.7	80.8	77.6	26.6	28.7	27.6	52.4	65.2	59.1
LocateAnything Slow NTP [ 51 ]	75.0	81.9	78.3	25.0	27.2	26.0	54.7	65.1	59.4

Table 4: Multi-instance grounding results on COCO 2017 [ 24 ] . Recall, precision, and F1 are reported at IoU thresholds of 0.50 and 0.95 and as mIoU aggregates. All scores are percentages. Best scores are bold; second-best scores are underlined.

Methods	ReasonSeg				RoboRefIt
	Val		Test		TestA		TestB
	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU	gIoU	cIoU
LISA ${}_{\text{LLaVA-7B}}$ [ 18 ]	52.9	54.0	47.3	48.4	36.1	29.9	28.7	22.7
LISA++ ${}_{\text{LLaVA-7B}}$ [ 64 ]	64.2	68.1	57.0	59.5	58.7	51.5	53.5	47.8
PixelLM ${}_{\text{LLaVA-7B}}$ [ 41 ]	44.7	37.4	44.0	39.7	45.2	38.4	38.2	31.4
Sa2VA ${}_{\text{InternVL2-2B}}$ [ 71 ]	59.2	60.0	56.9	55.8	56.8	29.0	34.0	10.2

Table 5: Reasoning segmentation on ReasonSeg [ 18 ] and embodied referring on RoboRefIt [ 28 ] . Scores are gIoU and cIoU (%, higher is better; Equation 15 ). Best scores are bold; second-best scores are underlined.

Methods	General Perception			High-Resolution Perception			Counting and Color
Methods	MME-RW-en	MME-RW-cn	CV-B	V*	HR-4K	HR-8K	CountQA	ColorB
Qwen2.5-VL ${}_{\text{7B}}$ [ 2 ]	60.8	58.3	75.3	78.5	71.6	67.9	18.9	76.4
MiMo-VL-RL ${}_{\text{7B}}$ [ 62 ]	63.4	59.8	84.3	81.2	74.4	72.9	28.3	82.8
Qwen3-VL ${}_{\text{4B}}$ [ 1 ]	63.5	63.6	85.0	80.1	78.3	72.9	28.1	81.6
Qwen3-VL ${}_{\text{8B}}$ [ 1 ]	66.0	66.7	85.4	86.4	78.9	74.6	29.0	82.8
MiniCPM-V-4.5 ${}_{\text{9B}}$ [ 70 ]	58.2	56.2	80.2	70.2	69.9	63.6	23.4	79.8

Table 6: General and fine-grained perception results. All scores are answer accuracy (%, higher is better). Benchmark names and abbreviations are defined in the evaluation setup. Best scores are bold; second-best scores are underlined.

Methods	Instruction				Domain
	Explicit		Implicit		General		Aerial		Driving		Panoramic
	cIoU	[email protected]	cIoU	[email protected]	cIoU	[email protected]	cIoU	[email protected]	cIoU	[email protected]	cIoU	[email protected]
GPT-5.6-Sol [ 35 ]	63.8	73.2	61.2	70.0	68.4	67.9	42.2	58.5	68.4	81.8	57.1	78.3
Gemini-3.6-Flash [ 11 ]	61.4	74.5	61.0	70.3	63.6	65.8	50.1	62.5	66.8	81.0	60.8	80.3
MiMo-V2.5 [ 63 ]	51.4	52.2	47.4	46.8	54.2	45.2	32.6	38.6	50.5	52.8	45.4	61.4
Seed-2.1-Pro [ 3 ]	62.1	70.2	55.9	64.5	60.6	56.7	50.1	58.5	66.8	77.4	59.9	76.7

Table 7: LensBench marginal results by instruction type and visual domain, pooling single and multiple targets. Scores are mask cIoU and box [email protected] (%, higher is better); [email protected] uses a box IoU threshold of 0.5. All models are evaluated by us with the same instructions and predict masks directly. Best scores are bold; second-best scores are underlined.

Figure 8: Qualitative comparison on LensBench across general, aerial, driving, and panoramic imagery. Each row gives the domain, target cardinality, and instruction type alongside the query. Columns show the input image, predictions from the compared models, and the ground truth (GT); red overlays indicate the selected regions. The examples illustrate attribute- and function-based target selection, recovery of multiple instances, and differences in mask extent.

Stage 2 Supervision		Referring Expression Comprehension			General Perception
BBox	Mask	RefCOCO	RefCOCO+	RefCOCOg	MME-RW-en	MME-RW-cn	CV-B
✗	✗	90.4	84.8	88.0	66.5	60.3	85.4
✓	✗	93.5	89.7	90.2	68.8	66.1	87.9
✓	✓	94.6	92.1	92.5	69.7	67.5	88.1

Table 8: Spatial supervision ablation on REC and general perception. Scores are validation-set [email protected] and answer accuracy (%), respectively. The first row is the Qwen3.5

{}_{\text{4B}}

backbone [ 37 ] .

Data		Referring Expression Comprehension			Referring Expression Segmentation
Grounding	Reasoning	RefCOCO	RefCOCO+	RefCOCOg	RefCOCO	RefCOCO+	RefCOCOg
✗	✗	90.4	84.8	88.0	—	—	—
✓	✗	94.5	91.5	92.0	82.7	79.1	80.6
✓	✓	94.6	92.1	92.5	83.4	79.9	81.1

Table 9: Training data ablation on REC and RES. Scores are validation-set [email protected] and cIoU (%), respectively. The first row is the Qwen3.5

{}_{\text{4B}}

backbone [ 37 ] ; — denotes unavailable results.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Box predictions for referring expressions based on appearance, action, relative size, and spatial relations. Each orange box marks the instance selected by the model.

Figure 10: Mask predictions for targets identified by referring expressions. Orange overlays delineate the selected fruit wedge, laptop, kite, and cow. The accompanying expressions specify the intended referent in each scene.

Figure 11: Multi-object box predictions in scenes with repeated instances and multiple categories. Labels identify the predicted category and distinguish instances within each category.

Figure 12: Predictions from functional descriptions. Each example pairs the instruction and response with the predicted boxes and masks in orange. The spacecraft and packaging traces explicitly consider multiple candidate referents.

Figure 13: Object selection from requests that combine category, appearance, and position. Orange boxes and masks show the predictions alongside the reasoning and response.

Figure 14: Region references in spatial comparison (top) and vehicle-color recognition (bottom). Matching colors link the response tokens to image regions. The enlarged view shows the small targets in the second example.

Figure 15: Instance-by-instance counting of climbing stones (top) and leaves (bottom). The traces contain five and six region references, respectively. Each colored token block corresponds to a box and a mask.

Large Vision-Language Models

Large vision-language models (LVLMs) connect image and video perception with language generation, reasoning, and interactive tasks.

See all 399 papers

Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Models
Sep 25, 2026Tianhang Guo, Yulin He, Wei Chen +3Reasoning Segmentation VLM Reasoning
National University of Defense Technology, Changsha, China
Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning
Jul 7, 2026Yake Wei, Yuan Wang, Fengyun Rao +2Vision-Language Grounding VLM Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China · Laboratory of Research on Large Models and Intelligent Governance · Tsinghua University +1
Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding
May 8, 2026Yuan Yao, Qiushi Yang, Humen Zhong +5Image Segmentation Referring Expression Segmentation
Tongyi Lab, Alibaba Group

Sep 25, 2026J/K move · Enter open · S save

Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Models

Tianhang Guo, Yulin He, Wei Chen, Wenjuan Zhou+2

National University of Defense Technology, Changsha, China

Reasoning Segmentation VLM Reasoning Latent Visual Reasoning Multimodal Large Language Models

Open paper AskPDF arXiv

Reasoning segmentation aims to interpret implicit textual queries and enable fine-grained visual perception, which is critical for applications such as human-computer interaction and embodied agents. Existing methods typically generate explicit Chain-of-Thought (CoT) by multimodal large language models (MLLMs) before localizing the target. Although intuitive, such explicit verbal reasoning introduces substantial attention interference: redundant textual tokens disrupt attention during perception-token generation and also increase the effective distance between visual tokens. To address this issue, we propose LIRSeg, which fully replaces explicit CoT with a compact set of learnable latent tokens for reasoning segmentation. LIRSeg is trained in two stages: spatial alignment grounds the latent tokens in object-relevant visual evidence, and GRPO further optimizes them with segmentation rewards. To make these compact latent tokens more informative, we introduce three complementary mechanisms from an information perspective: extreme-advantage sampling for selecting informative training signals, decoupled exploration-stability updates for learning complementary representations, and latent diversity amplification for preventing representational collapse. Extensive experiments on benchmarks demonstrate that LIRSeg consistently improves both segmentation accuracy and reasoning efficiency. Compared with the VisionReasoner baseline, LIRSeg achieves absolute gIoU improvements of 4.9% on ReasonSeg, 7.1% on MUSE, and 4.7% on MMR, while achieving a approximately 16x reduction in reasoning tokens. Code is available in supplementary materials.

Abstract

Large Vision-Language Models

Explore similar work

Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Models

Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning

Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding