Event-SAMA: Language-Grounded Video Object Segmentation on Neuromorphic Event Streams

Sergei Fedorchenko; Max Zehnder; Nikola Zubić; Rong Zou; Davide Scaramuzza

cs.LGOct 29, 2026

Event-SAMA: Language-Grounded Video Object Segmentation on Neuromorphic Event Streams

Authors: Sergei Fedorchenko, Max Zehnder, Nikola Zubić, Rong Zou, Davide Scaramuzza

Organizations: University of Zurich

Abstract

Event cameras offer microsecond temporal resolution, high dynamic range and no motion blur, but referring video object segmentation (RVOS) lacks event-stream datasets and models. We introduce Event-SAMA-239K, a synthetic-event conversion of the SAMA-239K grounded video-chat corpus and all additional SAMA training data: 30,650 video clips and 392,162 images. Optical-flow, FILM or saccade upsampling followed by a linear event generator preserves all mask, referring-expression and dialogue annotations. We also introduce Event-SAMA, adapting SAMA-4B with a two-channel polarity patch embedding initialised from pretrained RGB weights and a zero-initialised event adapter. End-to-end training yields 24.37 $\mathcal{J}\&\mathcal{F}$ on MeViS valid_u and 22.33 $\mathcal{J}\&\mathcal{F}$ on the converted ReVOS valid subset, establishing the first event-native RVOS baseline (RGB SAMA-4B: 55.4 on MeViS). Frozen RGB SAMA-4B with pseudo-RGB renderings or E2VID reconstructions of the same events scores 13.23 and 15.94 $\mathcal{J}\&\mathcal{F}$ . Without retraining, Event-SAMA works on real event-camera recordings when the accumulation window matches their event density. A seed-controlled ablation with a measured noise floor of 0.47 $\mathcal{J}\&\mathcal{F}$ finds gains from diverse training sources over MeViS alone (+4.99), RGB-mean event-stem initialisation (+4.33), and a trainable SAM2 mask decoder (+2.76); the event adapter, partial ViT unfreezing and event polarity are within noise at this budget. An earlier, partially converted corpus snapshot trained with the identical recipe scores 1.38 $\mathcal{J}\&\mathcal{F}$ higher on MeViS and 1.12 higher on ReVOS. Both are single runs that also differ in schedule length, so we draw no conclusion about corpus scale.

Figures & tables

Dataset	Type	Upsampling	$c_{\pm}$	Count	ev/px/s
Event-SAMA-239K core (masks, referring expressions, dialogue)
VidSTG	Video	optical flow	3	6,618	26.6
SA-V	Video	optical flow	5	2,867	20.8
LV-VIS	Video	optical flow	3	2,595	9.6
Ref-YouTube-VOS	Video	optical flow	5	1,976	55.1
MeViS	Video	optical flow	4	1,087	6.4

Table 1: Datasets converted to synthetic event streams. Count is video clips for video datasets and images for image datasets (annotated).

c_{\pm}

is the contrast threshold of the linear event generator. ev/px/s is the median number of events per pixel per second (2 s cap) over 200 randomly sampled items per dataset. † ReVOS comprises TAO (445 clips), OVIS (140) and UVO (77).

Panel of Figure 1 — Figure 1: Event-SAMA, adapted from SAMA-4B. Each event frame $E_{t}$ drives two paths. On the semantic/language path , the two-channel polarity stem (trained) feeds the frozen InternViT; per-keyframe features $F_{\mathrm{key}}$ pass through the pixel-shuffle/ mlp1 projection and the trained event adapter, while whole-clip context $F_{\mathrm{agg}}$ is produced by the trained STC aggregator. Both are interleaved with the text prompt and fed to the frozen Qwen2.5-3B LLM, which emits a [SEG] token. On the grounding path , $E_{t}$ is rendered to pseudo-RGB for the frozen SAM2 image encoder; the trained SAM2 mask decoder, prompted by the [SEG] hidden state, outputs the per-frame mask.

Model	Training data	$\mathcal{J}$	$\mathcal{F}$	MeViS $\mathcal{J}\&\mathcal{F}$	ReVOS $\mathcal{J}\&\mathcal{F}^{\ddagger}$
SAMA-4B (RGB, upper bound)	SAMA mix (RGB)	—	—	55.4	—
SAMA-4B (frozen)	pseudo-RGB events	12.42	14.03	13.23	—
SAMA-4B (frozen)	E2VID reconstr.	15.62	16.25	15.94	—
Event-SAMA	5-source subset (27,461)	25.32	26.19	25.75	23.45
Event-SAMA	Full corpus (422,812)	23.81	24.92	24.37	22.33

Table 2: Referring VOS on MeViS valid_u and ReVOS valid ‡ . Both Event-SAMA rows use the same 4-epoch recipe and differ only in training data. The full-corpus model (bold) is our main result because it is the model trained on the corpus described in Section 3 . The RGB SAMA-4B number is the published MeViS result ( Sun et al., 2025 ) ; it serves as an upper bound, not as a like-for-like comparison. The two reconstruct-and-reuse rows feed pseudo-RGB renderings or E2VID ( Rebecq et al., 2021 ) reconstructions of the same events to the unmodified RGB SAMA-4B with its own evaluation script.

Configuration	$\mathcal{J}$	$\mathcal{F}$	$\Delta\,\mathcal{J}\&\mathcal{F}$
Full model (baseline, mean of 2 seeds)	23.42	25.46	—
Effects outside the noise floor
$-$ diverse data mix (train on MeViS only)	18.19	20.71	$-4.99$
$-$ RGB-mean stem init (random init) †	16.79	20.63	$-4.33$
$-$ trainable SAM2 decoder (frozen)	20.65	22.70	$-2.76$
Effects within the noise floor

Table 3: Ablations on MeViS valid_u , 1-epoch iso-compute recipe on the five-source subset. The baseline is the mean of two seeds (24.20 / 24.67

\mathcal{J}\&\mathcal{F}

; spread 0.47). Rows in the first group lie outside the noise floor; rows in the second group do not, and the adapter and ViT variants reversed sign (

-0.57

+0.26

) on an independent earlier batch, whereas the decoder effect kept its sign in both batches. The last row is not iso-compute with the others (see footnote).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Event-SAMA on MeViS valid_u for the expression “elephants walking around”. Left: the RGB source frame, shown for reference only; the model never sees it. Right: the two-channel event frame the model receives, with the predicted mask overlaid. Only the moving elephant generates events; the static background is almost empty.

Keyframes $K$	8	16	32	Window $w$	5 ms	10 ms	20 ms
$\mathcal{J}\&\mathcal{F}$	20.64	21.93	23.04	$\mathcal{J}\&\mathcal{F}$	18.73	23.04	24.59

Appendix

Table 4: Protocol sweeps on a fixed checkpoint (MeViS valid_u ,

\mathcal{J}\&\mathcal{F}

). Left: number of keyframes

K

at a 10 ms accumulation window. Right: accumulation window

w

K=32

. Accuracy saturates in

K

but is strongly asymmetric in

w

Figure 3: Event-SAMA on real MouseSIS events (75 ms window), selected examples from two validation clips. Background: the events exactly as the grounding branch receives them; red: predicted mask; yellow: ground-truth contour. Top: “the mice” ; bottom: the absent “a red car” (empty predictions). Aggregate numbers are in Section 5.3 .

Figure 4: Distribution of events per pixel per second for converted ChatUniVi clips (200 clips) and for real Gen1 event-camera recordings [ de Tournemire et al., 2020 ] . Medians: ChatUniVi 4.9, Gen1 8.1 ev/px/s.

Figure 5: Converted ChatUniVi clip spanning 3 s. Top row: three consecutive source RGB frames. Bottom row: converted event accumulations over 100 ms windows covering the original and FILM-upsampled frames. Only moving regions generate events; the static background is silent.

Figure 6: Converted VidSTG clip spanning 0.2 s. Top row: two consecutive source RGB frames. Bottom row: converted event accumulations over 100 ms windows covering the original and optical-flow-upsampled frames.

Panel of Figure 7 — Figure 7: Saccade conversion of a still image. Left: source RGB image from LLaVA-1.5. Right: ON/OFF event accumulation after the $\pm 8$ px saccade (ON red, OFF blue). The saccade fires events across the full image extent, providing training signal that a static image would otherwise not produce.

Figure 8: Distribution of events per pixel per second for saccade-converted LLaVA-1.5 images (200 images) and for real Gen1 recordings [ de Tournemire et al., 2020 ] . Medians: LLaVA-1.5 40.8, Gen1 8.1 ev/px/s.

Video Object Segmentation

Video object segmentation assigns pixel-level masks to objects across frames while preserving their identities through motion, occlusion, and change.

See all 115 papers

Event-Driven Refresh and Recurrence Memory to Reduce Stale Grounding in Referring Video Object Segmentation
Sep 30, 2026Abu Hanif Muhammad Syarubany, Jaehyun Jang, Siwoo Lim +2Video Object Segmentation Referring Video Object Segmentation
School of Electrical Engineering, Korea Advanced Institute of Science & Technology (KAIST), Daejeon 34141, Republic of Korea
Key-Frame Reasoning with SAM3: Third Place Solution for the MeViS-Text Track of the 8th LSVOS Challenge
Aug 18, 2026Ce Bian, Xusheng He, Jinrong Zhang +3Referring Video Object Segmentation Video Object Segmentation
Harbin Institute of Technology, Shenzhen · Nanyang Technological University · Shenzhen Loop Area Institute
The 1st Winner for 5th PVUW MeViS-Text Challenge: Strong MLLMs Meet SAM3 for Referring Video Object Segmentation
Apr 1, 2026Xusheng He, Canyang Wu, Jinrong Zhang +3Referring Video Object Segmentation
Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, China

Sep 30, 2026J/K move · Enter open · S save

Event-Driven Refresh and Recurrence Memory to Reduce Stale Grounding in Referring Video Object Segmentation

Abu Hanif Muhammad Syarubany, Jaehyun Jang, Siwoo Lim, Seungyeon Ryu+1

School of Electrical Engineering, Korea Advanced Institute of Science & Technology (KAIST), Daejeon 34141, Republic of Korea

Video Object Segmentation Referring Video Object Segmentation

Open paper AskPDF arXiv

Referring Video Object Segmentation (RVOS) aims to produce a pixel-accurate mask sequence for an object specified by natural language. Sa2VA combines a multimodal large language model with SAM2 for grounded segmentation; however, its inference typically grounds the query from a small fixed set of initial keyframes and then relies on propagation. In long or dynamic videos, this can cause stale grounding and persistent false positives when the object composition changes (e.g., distractors enter or the target disappears/re-appears). We propose Event-Driven Refresh + Recurrence Memory (EDRRM), an enhancement that selectively re-invokes Sa2VA only at stable change points. EDRRM triggers refresh boundaries using an EMA-smoothed event score computed from tracking-derived cues (births/deaths and coarse composition/layout changes) with temporal constraints. A recurrence memory further retrieves anchor frames via CLIP similarity to re-condition the model on re-appearance events. Experiments on Ref-DAVIS17, MeViS, and ReVOS show that EDRRM achieves a competitive accuracy-efficiency trade-off relative to fixed-window and FrameDiff-SSIM baselines, maintaining comparable or superior J&F scores at substantially lower average refresh-call budgets and reducing false-positive failures. End-to-end runtime analysis further confirms that the overhead introduced by tracking, CLIP-based recurrence matching, and the identifiability gate remains modest relative to the dominant Sa2VA inference cost, thereby validating the efficiency of the proposed pipeline.

Abstract

Video Object Segmentation

Explore similar work

Event-Driven Refresh and Recurrence Memory to Reduce Stale Grounding in Referring Video Object Segmentation

Key-Frame Reasoning with SAM3: Third Place Solution for the MeViS-Text Track of the 8th LSVOS Challenge

The 1st Winner for 5th PVUW MeViS-Text Challenge: Strong MLLMs Meet SAM3 for Referring Video Object Segmentation