Eye image segmentation using visual and concept prompts with Segment Anything Model 3 (SAM3)

Diederick C. Niehorster; Marcus Nyström

cs.CVMar 18, 2026

Eye image segmentation using visual and concept prompts with Segment Anything Model 3 (SAM3)

Authors: Diederick C. Niehorster, Marcus Nyström

Organizations: Lund University Humanities Lab & Dept. of Psychology, Lund University, Lund, Sweden · Lund University Humanities Lab, Lund, Sweden

Abstract

Previous work has reported that vision foundation models show promising zero-shot performance in eye image segmentation. Here we examine whether the latest iteration of the Segment Anything Model, SAM3, offers better eye image segmentation performance than SAM2, and explore the performance of its new concept (text) prompting mode. Eye image segmentation performance was evaluated using diverse datasets encompassing both high-resolution high-quality videos from a lab environment and the TEyeD dataset consisting of challenging eye videos acquired in the wild. Results show that in most cases SAM3 with either visual or concept prompts did not perform better than SAM2, for both lab and in-the-wild datasets. Since SAM2 not only performed better but was also faster, we conclude that SAM2 remains the best option for eye image segmentation. We provide our adaptation of SAM3's codebase that allows processing videos of arbitrary duration.

Figures & tables

Figure 1. Example prompts and resulting segmentations for the high resolution lab datasets (left) and the TEyeD datasets (right) for SAM 2 visual prompting (top), SAM 3 visual prompting (middle) and SAM 3 concept prompting (bottom). For the lab images, the prompted frame, the 100th frame and the 1000th frame in the eye video are shown. For the TEyeD images, the prompted frame, the 1000th, the 10000th and the 20000th frame are shown. For the lab datasets (left): the dark blue mask indicates the CR, green the pupil, red the iris and cyan the sclera. For the TEyeD datasets (right): dark blue is the pupil, green the iris, red the sclera. Positive (+) and negative (square) prompts use the same color codes. For the bottom row, different colors instead indicate different “pupil” objects returned by the model. The brown in some segmentations results from the iris and sclera masks overlapping. Close-up eye images with different segmentations of parts of the eye.

Figure 2. Prompting TEyeD. The left panel shows the ground truth annotations provided by TEyeD for the pupil (green ellipse), iris (red ellipse) and palpebral fissure (yellow polygon). Also indicated are the determined eye corners (cyan dots), closest points on the iris ellipse (orange points) and the derived prompt coordinates for the pupil (green +), the iris (red +) and the sclera (blue and magenta +). The right panel shows the corresponding positive (+) and negative (square) visual prompts provided to both SAM models (blue: pupil, green: iris, red: sclera). Also shown are the output masks from SAM 3, using the same color codes as the prompts. Close-up eye images with ground truth annotation and output segmentation of parts of the eye.

Figure 3. RMS-S2S precision and data loss for SAM 2 and SAM 3 on the high-resolution lab datasets. Shown are the RMS-S2S precision (lower values is better) and data loss (lower is better) per participant. Note that the range of the y-axis is different for each of the panels. A summary showing the precision or data loss on the same scale is shown in the right most bar graphs (error bars indicate SEM across participants). Stars indicate significant differences according to paired t-tests. A collection of line and bar graphs.

Dataset	mIoU				Overlap	FA rate			Miss rate			Youden’s J
	pupil	iris	sclera	eye opening	iris–sclera	Pupil	Iris	Sclera	Pupil	Iris	Sclera	Pupil	Iris	Sclera
SAM 2 visual
Dikablis	0.880	0.814	$0.495$	0.773	$0.270$	0.065	0.384	0.377	$0.037$	$0.003$	$0.011$	0.898	0.613	0.612
GiW	0.911	0.768	$0.464$	0.767	$0.483$	0.193	0.340	0.373	0.010	$0.032$	$0.050$	0.798	0.628	0.578
LPW	0.853	$0.692$	$0.333$	$0.639$	$0.587$	0.506	0.668	0.851	0.023	$0.055$	$0.073$	0.470	0.278	$0.076$
NVGaze	0.895	0.854	$0.593$	0.816	$0.227$	0.195	0.447	0.548	0.011	$0.008$	$0.009$	0.794	0.545	0.443

Table 1. Results for SAM 2 and SAM 3 per dataset in the TEyeD set. Shown are mIoU for the pupil, iris, sclera and the whole eye opening. Furthermore shown is the overlap between the iris and the sclera, and the false alarm rate, miss rate and Youden’s J for the pupil, iris and sclera. For each column and dataset, the best values are printed in bold (higher is better for mIoU and Youden’s J, while lower is better for the other columns), and for pupil columns the second best value is underlined.

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 768 papers

Is SAM3 ready for pathology segmentation?
Apr 20, 2026Qiuyu Kong, Shakiba Sharifi, Yiming Wang +2Computational Pathology Image Segmentation
Sapienza University of Rome · University of Verona · Fondazione Bruno Kessler
TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision
Oct 10, 2026Hao Zhu, Heyu Sun, Shuo Jin +2Promptable Image Segmentation Open-Vocabulary Semantic Segmentation
PicoSAM3: Real-Time In-Sensor Region-of-Interest Segmentation
Mar 12, 2026Pietro Bonazzi, Nicola Farronato, Stefan Zihlmann +2In-Sensor Computing Promptable Image Segmentation
Equal contribution · ETH Zürich, Zürich, Switzerland. · IBM Research, Zürich, Switzerland.

Apr 20, 2026J/K move · Enter open · S save

Is SAM3 ready for pathology segmentation?

Qiuyu Kong, Shakiba Sharifi, Yiming Wang, Marco Cristani+1

Sapienza University of Rome · University of Verona · Fondazione Bruno Kessler

Computational Pathology Image Segmentation Histopathology Image Segmentation Promptable Image Segmentation Medical Image Segmentation Few-Shot Medical Image Segmentation

Open paper AskPDF arXiv

Is Segment Anything Model 3 (SAM3) capable in segmenting Any Pathology Images? Digital pathology segmentation spans tissue-level and nuclei-level scales, where traditional methods often suffer from high annotation costs and poor generalization. SAM3 introduces Promptable Concept Segmentation, offering a potential automated interface via text prompts. With this work, we propose a systematic evaluation protocol to explore the capability space of SAM3 in a structured manner. Specifically, we evaluate SAM3 under different supervision settings including zero-shot, few-shot, and supervised with varying prompting strategies. Our extensive evaluation on pathological datasets including NuInsSeg, PanNuke and GlaS, reveals that: (1) text-only prompts poorly activate nuclear concepts; (2) performance is highly sensitive to visual prompt types and budgets; (3) few-shot learning offers gains, but SAM3 lacks robustness against visual prompt noise; and (4) a significant gap persists between prompt-based usage and task-trained adapter-based reference. Our study delineates SAM3's boundaries in pathology image segmentation and provides practical guidance on the necessity of pathology domain adaptation.

Abstract

Medical Image Segmentation

Explore similar work

Is SAM3 ready for pathology segmentation?

TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision

PicoSAM3: Real-Time In-Sensor Region-of-Interest Segmentation