Adapting Open-Weight MLLMs to Generate Point Prompts for Electron Microscopy Segmentation

Samia Mohinta; Albert Cardona

cs.CVSep 12, 2026

Adapting Open-Weight MLLMs to Generate Point Prompts for Electron Microscopy Segmentation

Authors: Samia Mohinta, Albert Cardona

Organizations: University of Cambridge, Cambridge, UK · MRC Laboratory of Molecular Biology, Cambridge, UK

Abstract

Promptable models such as microSAM segment electron microscopy (EM) images from point prompts, but automation requires generating prompts without user input. We ask whether open-weight multimodal large language models (MLLMs) can generate them from natural-language requests by returning coordinates to a frozen segmenter. To that end, we convert masks from three mitochondria datasets into training examples, pairing images and instructions with centroid coordinates, then train LoRA adapters while freezing the MLLM backbone and microSAM. We find that Qwen3-VL reaches segmentation AP $_{50}$ $0.736$ after supervised fine-tuning and reward optimization, up from $0.247$ without adaptation, while automatic prompt generation (APG) achieves $0.773$ . In addition, two other MLLMs improve, reaching or exceeding APG. When compared with a supervised centroid-heatmap detector that reaches AP $_{50}$ $0.904$ for this mitochondria task, Qwen3-VL more closely matches the annotated point set and instance counts. Moreover, training on two public datasets transfers to an unseen third, while training on all three transfers to an independent EM volume. Robustness tests show stable performance under unseen formulations of the natural-language request, while the coordinates can be reused by a second segmenter. To our knowledge, this is the first feasibility study of open-weight MLLMs as EM point generators, providing an inspectable, language-directed link between localization and mask decoding.

Figures & tables

Figure 1: Mask-derived supervision turns an open-weight MLLM into an inspectable point generator for a frozen segmenter. Segmentation masks are reduced to centroid targets that train a low-rank adapter on Qwen3-VL [ 3 ] . Given an EM crop and a natural-language request, the adapted model emits coordinates that the frozen microSAM decodes into instance masks.

Figure 2: Point localization, point-source diagnostics, and downstream segmentation. (a) Qwen3-VL point F1 at $\tau=0.05$ . Lucchi++, VNC, and MitoEM-H show evaluation on the individual datasets; the SFT and SFT+GRPO bars use adapters trained separately on each dataset. “All data” uses the combined adapter. (b) Point-source diagnostics for Qwen3-VL SFT+GRPO and the supervised centroid-heatmap detector on “All data”. (c) Instance AP 50 over the same test set. Point sources are decoded by frozen microSAM; the U-Net predicts masks directly. Qwen3-VL SFT and SFT+GRPO use the seed-42 run with three SFT epochs, whereas InternVL3.5 and Molmo2 use one. Figure 5 d gives the matched one-epoch comparison. Oracle centroids prompt every annotated instance at its ground-truth centroid.

MLLM point generators
Prompt source	Training	AP 50 $\uparrow$	AP 75 $\uparrow$	mAP $\uparrow$	Dice $\uparrow$	mSA $\uparrow$	MAE $\downarrow$
Qwen3-VL	zero-shot	0.247	0.235	0.202	0.323	0.183	1.182
Qwen3-VL	SFT	0.712	0.659	0.570	0.776	0.538	0.606
Qwen3-VL	SFT+GRPO	0.736	0.678	0.585	0.792	0.547	0.564
InternVL3.5	SFT	0.774	0.714	0.613	0.812	0.574	0.535
Molmo2	SFT	0.817	0.747	0.644	0.836	0.611	0.461

Table 1: End-to-end results on “All data”. Point generators are decoded by frozen microSAM; the combined U-Net predicts masks directly. Qwen3-VL SFT and SFT+GRPO use the primary seed-42 run with three SFT epochs, whereas InternVL3.5 and Molmo2 use one SFT epoch. Best MLLM values are in bold ; best Qwen3-VL values are underlined .

Figure 3: Point-to-mask hand-off on selected crops. Rows show Lucchi++, VNC, and MitoEM-H; columns show the raw EM image, ground truth, Qwen3-VL SFT point predictions, masks decoded from those points, APG masks, and masks decoded from centroid-heatmap points. Colors distinguish instances; point markers indicate the prompts supplied to microSAM. Cyan arrowheads indicate additional centroid-heatmap point predictions that do not match annotated instances.

Figure 4: Decoder portability, instruction sensitivity, and independent-volume transfer. (a) The same saved Qwen3-VL SFT points decoded by frozen microSAM and SAM 3. (b) AP 50 under the standard instruction and five held-out mitochondria phrasings for the existing SFT and SFT+GRPO checkpoints. (c) Standard and morphology-only instructions for unadapted Qwen3-VL and the existing SFT checkpoint; dashed lines show the mean over the three morphology-only phrasings. (d) Point-in-mask precision, object recall, and AP 50 for the unchanged combined three-epoch Qwen3-VL SFT adapter on “All data” and the independent larval-fly volume.

Figure 5: Training data, transfer, and cross-backbone adaptation. (a) Qwen3-VL SFT and SFT+GRPO AP 50 on the fixed Lucchi++ test set as training data increase. (b) Matched-size VNC analysis using one-epoch Qwen3-VL SFT and

433

unique training crops from VNC only, a three-dataset mixture, or Lucchi++ and MitoEM-H without VNC. Bars show mean AP 50 across three seeds; error bars show standard deviation. (c) Leave-one-dataset-out AP 50 (bars) and point F1 (lines) for separately trained Qwen3-VL and centroid-heatmap point generators. (d) Zero-shot and matched one-epoch SFT results across the three MLLM backbones; the dashed line marks APG AP 50 .

Figure 6: Adaptation and backend ablations. (a) Mean

\pm

standard deviation across seeds

7

42

, and

123

for Qwen3-VL SFT and SFT+GRPO. (b) AP 50 across tested LoRA ranks. (c) The same Qwen3-VL GRPO points decoded with the vit_b and vit_l microSAM backends.

Setting	SFT	GRPO
LoRA rank / $\alpha$ / dropout	$32/64/0.05$	inherited from SFT
Learning rate	$2\times 10^{-5}$	$5\times 10^{-7}$
Per-device batch size	$1$	$1$
Gradient accumulation	$8$	$4$
Epochs	$3$ primary; $1$ cross-model	$1$
Schedule / warmup	cosine / $0.05$	implementation default

Table 2: Core training settings used for the MLLM experiments. The primary Qwen3-VL result uses three SFT epochs; the matched cross-backbone comparison uses one epoch for every model.

Dataset	Training crops	Test crops
Lucchi++	2,869	342
VNC	433	79
MitoEM-H	2,655	332
Combined	5,957	753

Table 3: Training and test crop counts for the three public datasets.

Model and stage	Diag. F1 @0.10	Point-in-mask prec.	Object recall	Point-count MAE	Centroids in masks	Instances
Qwen3-VL zero-shot	0.170	0.293	0.201	1.182	0.988	1747/1768
Qwen3-VL SFT	0.747	0.852	0.718	0.603	0.988	1747/1768
Qwen3-VL SFT+GRPO	0.765	0.854	0.742	0.566	0.988	1747/1768
InternVL3.5 SFT	0.806	0.897	0.791	0.534	0.988	1747/1768
Molmo2 SFT	0.846	0.918	0.840	0.465	0.988	1747/1768

Table 4: Crop-aligned point diagnostics over the

753

“All data” test crops. Diagnostic point F1 uses threshold

0.10

; point-count MAE uses valid parsed point tokens.

Stage	Lucchi++	MitoEM-H	VNC	Macro	All data
SFT	0.831	0.605	0.647	0.695	0.712
SFT+GRPO	0.843	0.630	0.714	0.729	0.736

Table 5: The combined Qwen3-VL model evaluated by source condition. Macro is the unweighted mean of the three conditions; “All data” is the average over all

753

test crops. Values are AP 50 .

Dataset	Crops	Instances	Mean inst./crop	Median inst./crop	Median area (px)
Lucchi++	342	715	2.091	2	2,109
MitoEM-H	332	899	2.708	3	1,108
VNC	79	154	1.949	2	1,390

Table 6: Ground-truth instance density and size on the public test crops. Instance area is measured in pixels within the

256\times 256

crops.

LoRA rank	AP 50
4	0.565
8	0.599
16	0.619
32	0.712
64	0.653

Table 7: Combined Qwen3-VL SFT sensitivity to LoRA rank, same training data and

753

test crops through frozen vit_b_em_organelles . Values are AP 50 .

Input source	Backend	Dice	AP 50	AP 75	mAP	mSA	Count MAE
SAM 3 text prompt	SAM 3	0.008	0.008	0.008	0.008	0.008	2.348
Qwen3-VL SFT points	SAM 3	0.761	0.698	0.642	0.564	0.536	0.607
Qwen3-VL SFT points	microSAM	0.776	0.712	0.659	0.570	0.538	0.606

Table 8: Backend-portability control on the same

753

“All data” test crops. The two Qwen3-VL rows use identical saved SFT point predictions and differ only in the frozen downstream decoder. Direct SAM 3 prompting uses the text prompt mitochondrion .

Subset	Crops	Diag. F1	Point-in-mask	Object recall	Dice	AP 50	AP 75	mAP	mSA	Count MAE
Larval-fly 1%	131	0.545	0.753	0.462	0.567	0.464	0.438	0.379	0.372	1.214

Table 9: Independent-volume evaluation on the fixed one-percent in-house larval-fly subset.

Backend	Dice	AP 50	AP 75	mAP	mSA	Count MAE
vit_b_em_organelles	0.792	0.736	0.678	0.585	0.547	0.564
vit_l_em_organelles	0.797	0.729	0.686	0.601	0.569	0.564

Table 10: Backend-capacity control using the same saved Qwen3-VL SFT+GRPO point predictions on all

753

test crops.

Stage	Phrasing	Dice $\uparrow$	AP 50 $\uparrow$	mAP $\uparrow$	Count MAE $\downarrow$
SFT	Standard	0.776	0.712	0.570	0.606
SFT	pv01	0.777	0.715	0.571	0.603
SFT	pv02	0.782	0.726	0.580	0.579
SFT	pv03	0.775	0.712	0.570	0.614
SFT	pv04	0.785	0.723	0.576	0.575
SFT	pv05	0.778	0.717	0.571	0.620

Table 11: Sensitivity to five held-out formulations of the mitochondria phrasings. All rows use the same

753

test crops and frozen vit_b_em_organelles backend.

	Unadapted		SFT checkpoint
Phrasing	Point F1	AP 50	Point F1	AP 50
Morph-1	0.036	0.180	0.664	0.606
Morph-2	0.053	0.287	0.713	0.730
Morph-3	0.042	0.276	0.707	0.668
Mean	0.044	0.248	0.695	0.668

Table 12: Morphology-only phrasing control on “All data” at inference. Point F1 uses

\tau=0.05

Dataset	SFT	SFT+GRPO
Lucchi++	0.709	0.711
MitoEM-H	0.531	0.567
VNC	0.436	0.463

Table 13: Qwen3-VL adapters trained and tested separately on each dataset using seed

42

and three SFT epochs. SFT+GRPO adds one GRPO epoch. Values are AP 50 .

Setting	Point F1	PiM	Obj. recall	AP 50	mSA	Count MAE
Combined	0.674	0.655	0.964	0.904	0.456	1.483
Held-out Lucchi++	0.424	0.383	0.920	0.888	0.282	3.570
Held-out VNC	0.385	0.325	0.884	0.804	0.194	3.722
Held-out MitoEM-H	0.549	0.660	0.749	0.708	0.387	1.108

Table 14: Additional centroid-heatmap diagnostics. Point F1 uses

\tau=0.05

Condition	Crops	Dice	AP 50	AP 75	mAP	mSA	Count MAE
Lucchi++	342	0.949	0.944	0.934	0.890	0.837	0.231
MitoEM-H	332	0.865	0.834	0.741	0.682	0.560	0.756
VNC	79	0.824	0.863	0.693	0.637	0.490	0.835
All data	753	0.899	0.887	0.824	0.772	0.679	0.526

Table 15: The single combined U-Net evaluated by source condition. “All data” is the average over all

753

test crops.

Radius	Point F1	AP 50	Count MAE
8	0.674	0.904	1.483
12	0.677	0.904	1.466
16	0.679	0.904	1.434
24	0.681	0.903	1.378
32	0.683	0.900	1.303
48	0.690	0.885	1.094

Table 16: Peak-NMS sensitivity of the combined centroid-heatmap detector on “All data.” The detector is not retrained. Radius

8

is the setting used for the reported result. Point F1 uses

\tau=0.05

Figure 7: Sensitivity of the leave-one-dataset-out centroid-heatmap detectors to peak suppression. The minimum peak distance is varied for the three detectors;

8

pixels is the setting used for the reported transfer results. Point F1 uses the primary matching threshold

\tau=0.05

. Increasing suppression generally increases Point F1 and reduces instance-count MAE, while downstream AP 50 remains comparatively stable across the sweep.

Figure 8: Qualitative effect of heatmap peak suppression. Selected examples show the raw EM image, ground truth, Qwen3-VL SFT point predictions, and centroid-heatmap points and decoded masks at increasing peak-NMS minimum distances (radii: r=8, r=24, r=48). Stronger suppression removes nearby heatmap proposals (cyan arrowheads) while many of the corresponding decoded masks remain similar.

Point strategy	Mean selected-mask IoU	IoU $\geq.50$	IoU $\geq.75$	Centroid dist.	Boundary dist.
Projected centroid	0.828	0.954	0.833	0.21	16.80
Boundary-near	0.827	0.941	0.852	23.40	2.05
Distance maximum	0.819	0.943	0.825	9.37	20.88
Random interior	0.821	0.939	0.833	17.81	7.94

Table 17: Sensitivity of microSAM to valid point placement. Each strategy supplies one point for the same

1{,}768

ground-truth instances. Distances are measured in pixels.

Electron Microscopy

See all 36 papers

Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token
Mar 19, 2026Anqi Zhang, Xiaokang Ji, Guangyu Gao +3Image Segmentation Semantic Segmentation
Beijing Institute of Technology · University of Birmingham · Beijing Jiaotong University +1
Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation
Aug 3, 2026Jiazhen Liu, Mingkuan Feng, Long ChenInstance Segmentation Image Segmentation
The Hong Kong University of Science and Technology (HKUST), Hong Kong.
Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding
May 8, 2026Yuan Yao, Qiushi Yang, Humen Zhong +5Image Segmentation Referring Expression Segmentation
Tongyi Lab, Alibaba Group

Mar 19, 2026J/K move · Enter open · S save

Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token

Anqi Zhang, Xiaokang Ji, Guangyu Gao, Jianbo Jiao+2

Beijing Institute of Technology · University of Birmingham · Beijing Jiaotong University+1

Image Segmentation Semantic Segmentation Multimodal Large Language Models

Open paper AskPDF arXiv

Recent segmentation methods leveraging Multi-modal Large Language Models (MLLMs) have shown reliable object-level segmentation and enhanced spatial perception. However, almost all previous methods predominantly rely on specialist mask decoders to interpret masks from generated segmentation-related embeddings and visual features, or incorporate multiple additional tokens to assist. This paper aims to investigate whether and how we can unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E) while achieving competitive results, which eliminates the need for external decoders. To this end, our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs. First, we retain image features at their original uncompressed resolution, and refill them with residual features extracted from MLLM-processed compressed features, thereby improving feature precision. Subsequently, we integrate pixel-unshuffle operations on image features with and without LLM processing, respectively, to unleash the details of compressed features and amplify the residual features under uncompressed resolution, which further enhances the resolution of refilled features. Moreover, we redesign the attention mask with dual perception pathways, i.e., image-to-image and image-to-segmentation, enabling rich feature interaction between pixels and the segmentation token. Comprehensive experiments across multiple segmentation tasks validate that SELF1E achieves performance competitive with specialist mask decoder-based methods, demonstrating the feasibility of decoder-free segmentation in MLLMs. Project page: https://github.com/ANDYZAQ/SELF1E.

Abstract

Electron Microscopy

Explore similar work

Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token

Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation

Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding