cs.LGOct 29, 2026

Event-SAMA: Language-Grounded Video Object Segmentation on Neuromorphic Event Streams

Authors: Sergei Fedorchenko, Max Zehnder, Nikola Zubić, Rong Zou, Davide Scaramuzza

Organizations: University of Zurich

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Event cameras offer microsecond temporal resolution, high dynamic range and no motion blur, but referring video object segmentation (RVOS) lacks event-stream datasets and models. We introduce Event-SAMA-239K, a synthetic-event conversion of the SAMA-239K grounded video-chat corpus and all additional SAMA training data: 30,650 video clips and 392,162 images. Optical-flow, FILM or saccade upsampling followed by a linear event generator preserves all mask, referring-expression and dialogue annotations. We also introduce Event-SAMA, adapting SAMA-4B with a two-channel polarity patch embedding initialised from pretrained RGB weights and a zero-initialised event adapter. End-to-end training yields 24.37 J&F\mathcal{J}\&\mathcal{F} on MeViS valid_u and 22.33 J&F\mathcal{J}\&\mathcal{F} on the converted ReVOS valid subset, establishing the first event-native RVOS baseline (RGB SAMA-4B: 55.4 on MeViS). Frozen RGB SAMA-4B with pseudo-RGB renderings or E2VID reconstructions of the same events scores 13.23 and 15.94 J&F\mathcal{J}\&\mathcal{F}. Without retraining, Event-SAMA works on real event-camera recordings when the accumulation window matches their event density. A seed-controlled ablation with a measured noise floor of 0.47 J&F\mathcal{J}\&\mathcal{F} finds gains from diverse training sources over MeViS alone (+4.99), RGB-mean event-stem initialisation (+4.33), and a trainable SAM2 mask decoder (+2.76); the event adapter, partial ViT unfreezing and event polarity are within noise at this budget. An earlier, partially converted corpus snapshot trained with the identical recipe scores 1.38 J&F\mathcal{J}\&\mathcal{F} higher on MeViS and 1.12 higher on ReVOS. Both are single runs that also differ in schedule length, so we draw no conclusion about corpus scale.

Figures & tables

DatasetTypeUpsamplingc±c_{\pm}Countev/px/s
Event-SAMA-239K core (masks, referring expressions, dialogue)
VidSTGVideooptical flow36,61826.6
SA-VVideooptical flow52,86720.8
LV-VISVideooptical flow32,5959.6
Ref-YouTube-VOSVideooptical flow51,97655.1
MeViSVideooptical flow41,0876.4
Table 1: Datasets converted to synthetic event streams. Count is video clips for video datasets and images for image datasets (annotated). c±c_{\pm} is the contrast threshold of the linear event generator. ev/px/s is the median number of events per pixel per second (2 s cap) over 200 randomly sampled items per dataset. † ReVOS comprises TAO (445 clips), OVIS (140) and UVO (77).
Figure 1: Event-SAMA, adapted from SAMA-4B. Each event frame EtE_{t} drives two paths. On the semantic/language path , the two-channel polarity stem (trained) feeds the frozen InternViT; per-keyframe features FkeyF_{\mathrm{key}} pass through the pixel-shuffle/ mlp1 projection and the trained event adapter, while whole-clip context FaggF_{\mathrm{agg}} is produced by the trained STC aggregator. Both are interleaved with the text prompt and fed to the frozen Qwen2.5-3B LLM, which emits a [SEG] token. On the grounding path , EtE_{t} is rendered to pseudo-RGB for the frozen SAM2 image encoder; the trained SAM2 mask decoder, prompted by the [SEG] hidden state, outputs the per-frame mask.
ModelTraining dataJ\mathcal{J}F\mathcal{F}MeViS J&F\mathcal{J}\&\mathcal{F}ReVOS J&F‡\mathcal{J}\&\mathcal{F}^{\ddagger}
SAMA-4B (RGB, upper bound)SAMA mix (RGB)——55.4—
SAMA-4B (frozen)pseudo-RGB events12.4214.0313.23—
SAMA-4B (frozen)E2VID reconstr.15.6216.2515.94—
Event-SAMA5-source subset (27,461)25.3226.1925.7523.45
Event-SAMAFull corpus (422,812)23.8124.9224.3722.33
Table 2: Referring VOS on MeViS valid_u and ReVOS valid ‡ . Both Event-SAMA rows use the same 4-epoch recipe and differ only in training data. The full-corpus model (bold) is our main result because it is the model trained on the corpus described in Section 3 . The RGB SAMA-4B number is the published MeViS result ( Sun et al., 2025 ) ; it serves as an upper bound, not as a like-for-like comparison. The two reconstruct-and-reuse rows feed pseudo-RGB renderings or E2VID ( Rebecq et al., 2021 ) reconstructions of the same events to the unmodified RGB SAMA-4B with its own evaluation script.
ConfigurationJ\mathcal{J}F\mathcal{F}Δ J&F\Delta\,\mathcal{J}\&\mathcal{F}
Full model (baseline, mean of 2 seeds)23.4225.46—
Effects outside the noise floor
−- diverse data mix (train on MeViS only)18.1920.71−4.99-4.99
−- RGB-mean stem init (random init) †16.7920.63−4.33-4.33
−- trainable SAM2 decoder (frozen)20.6522.70−2.76-2.76
Effects within the noise floor
Table 3: Ablations on MeViS valid_u , 1-epoch iso-compute recipe on the five-source subset. The baseline is the mean of two seeds (24.20 / 24.67 J&F\mathcal{J}\&\mathcal{F} ; spread 0.47). Rows in the first group lie outside the noise floor; rows in the second group do not, and the adapter and ViT variants reversed sign ( −0.57-0.57 , +0.26+0.26 ) on an independent earlier batch, whereas the decoder effect kept its sign in both batches. The last row is not iso-compute with the others (see footnote).
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Event-SAMA on MeViS valid_u for the expression “elephants walking around”. Left: the RGB source frame, shown for reference only; the model never sees it. Right: the two-channel event frame the model receives, with the predicted mask overlaid. Only the moving elephant generates events; the static background is almost empty.
Keyframes KK81632Window ww5 ms10 ms20 ms
J&F\mathcal{J}\&\mathcal{F}20.6421.9323.04J&F\mathcal{J}\&\mathcal{F}18.7323.0424.59
Appendix
Table 4: Protocol sweeps on a fixed checkpoint (MeViS valid_u , J&F\mathcal{J}\&\mathcal{F} ). Left: number of keyframes KK at a 10 ms accumulation window. Right: accumulation window ww at K=32K=32 . Accuracy saturates in KK but is strongly asymmetric in ww .
Figure 3: Event-SAMA on real MouseSIS events (75 ms window), selected examples from two validation clips. Background: the events exactly as the grounding branch receives them; red: predicted mask; yellow: ground-truth contour. Top: “the mice” ; bottom: the absent “a red car” (empty predictions). Aggregate numbers are in Section 5.3 .
Figure 4: Distribution of events per pixel per second for converted ChatUniVi clips (200 clips) and for real Gen1 event-camera recordings [ de Tournemire et al., 2020 ] . Medians: ChatUniVi 4.9, Gen1 8.1 ev/px/s.
Figure 5: Converted ChatUniVi clip spanning 3 s. Top row: three consecutive source RGB frames. Bottom row: converted event accumulations over 100 ms windows covering the original and FILM-upsampled frames. Only moving regions generate events; the static background is silent.
Figure 6: Converted VidSTG clip spanning 0.2 s. Top row: two consecutive source RGB frames. Bottom row: converted event accumulations over 100 ms windows covering the original and optical-flow-upsampled frames.
Figure 7: Saccade conversion of a still image. Left: source RGB image from LLaVA-1.5. Right: ON/OFF event accumulation after the ±8\pm 8 px saccade (ON red, OFF blue). The saccade fires events across the full image extent, providing training signal that a static image would otherwise not produce.
Figure 8: Distribution of events per pixel per second for saccade-converted LLaVA-1.5 images (200 images) and for real Gen1 recordings [ de Tournemire et al., 2020 ] . Medians: LLaVA-1.5 40.8, Gen1 8.1 ev/px/s.
More in this topic

Video Object Segmentation

Video object segmentation assigns pixel-level masks to objects across frames while preserving their identities through motion, occlusion, and change.

See all 115 papers

Explore similar work

CardsList
Sep 30, 2026

Event-Driven Refresh and Recurrence Memory to Reduce Stale Grounding in Referring Video Object Segmentation

Referring Video Object Segmentation (RVOS) aims to produce a pixel-accurate mask sequence for an object specified by natural language. Sa2VA combines a multimodal large language model with SAM2 for grounded segmentation; however, its inference typically grounds the query from a small fixed set of initial keyframes and then relies on propagation. In long or dynamic videos, this can cause stale grounding and persistent false positives when the object composition changes (e.g., distractors enter or the target disappears/re-appears). We propose Event-Driven Refresh + Recurrence Memory (EDRRM), an enhancement that selectively re-invokes Sa2VA only at stable change points. EDRRM triggers refresh boundaries using an EMA-smoothed event score computed from tracking-derived cues (births/deaths and coarse composition/layout changes) with temporal constraints. A recurrence memory further retrieves anchor frames via CLIP similarity to re-condition the model on re-appearance events. Experiments on Ref-DAVIS17, MeViS, and ReVOS show that EDRRM achieves a competitive accuracy-efficiency trade-off relative to fixed-window and FrameDiff-SSIM baselines, maintaining comparable or superior J&F scores at substantially lower average refresh-call budgets and reducing false-positive failures. End-to-end runtime analysis further confirms that the overhead introduced by tracking, CLIP-based recurrence matching, and the identifiability gate remains modest relative to the dominant Sa2VA inference cost, thereby validating the efficiency of the proposed pipeline.

Abu Hanif Muhammad Syarubany, Jaehyun Jang, Siwoo Lim +2

School of Electrical Engineering, Korea Advanced Institute of Science & Technology (KAIST), Daejeon 34141, Republic of Korea

Video Object SegmentationReferring Video Object Segmentation
AskPDF
Aug 18, 2026

Key-Frame Reasoning with SAM3: Third Place Solution for the MeViS-Text Track of the 8th LSVOS Challenge

This report presents a two-stage, training-free solution for the MeViS-Text track of the 8th LSVOS Challenge. The task requires a model to localize and segment the object specified by a natural-language expression throughout a video. Such expressions often depend on temporal cues, including actions, interactions, directions, and relative positions. Our first stage uses Gemini-3.1 Pro via API to decompose a video-level event into instance-level targets, select a key frame for each target, and generate a discriminative description aligned with that frame. In the second stage, SAM3-agent produces a pixel-level seed mask on the selected frame, and the SAM3 video tracker propagates the mask bidirectionally through the video. Valid instances are grounded and propagated independently before their frame-wise masks are merged. All local SAM3 processing runs on a single NVIDIA GeForce RTX 4090 without task-specific training or model ensembling. Our method ranked third on the challenge test set, obtaining J&F, J, F, N-acc., T-acc., and Final scores of 0.761, 0.7367, 0.7852, 0.8333, 0.9755, and 0.856593, respectively.

Ce Bian, Xusheng He, Jinrong Zhang +3

Harbin Institute of Technology, Shenzhen · Nanyang Technological University · Shenzhen Loop Area Institute

Referring Video Object SegmentationVideo Object SegmentationVisual Object Tracking
AskPDF
Apr 1, 2026

The 1st Winner for 5th PVUW MeViS-Text Challenge: Strong MLLMs Meet SAM3 for Referring Video Object Segmentation

This report presents our winning solution to the 5th PVUW MeViS-Text Challenge. The track studies referring video object segmentation under motion-centric language expressions, where the model must jointly understand appearance, temporal behavior, and object interactions. To address this problem, we build a fully training-free pipeline that combines strong multimodal large language models with SAM3. Our method contains three stages. First, Gemini-3.1 Pro decomposes each target event into instance-level grounding targets, selects the frame where the target is most clearly visible, and generates a discriminative description. Second, SAM3-agent produces a precise seed mask on the selected frame, and the official SAM3 tracker propagates the mask through the whole video. Third, a refinement stage uses Qwen3.5-Plus and behavior-level verification to correct ambiguous or semantically inconsistent predictions. Without task-specific fine-tuning, our method ranks first on the PVUW 2026 MeViS-Text test set, achieving a Final score of 0.909064 and a J&F score of 0.7897. The code is available at https://github.com/Moujuruo/MeViSv2_Track_Solution_2026.

Xusheng He, Canyang Wu, Jinrong Zhang +3

Harbin Institute of Technology, Shenzhen, China · Shenzhen Loop Area Institute, China

Referring Video Object Segmentation
AskPDF