Sekai: Incremental Event Memory for Generalized Reasoning Video Segmentation
Organizations: University of Science and Technology of China · National University of Singapore
Abstract
Online reasoning video segmentation (Online RVS) requires a model to segment query-referred objects as a video streams in, using only the frames observed so far. Existing formulations largely assume a fixed target set throughout the video, overlooking a fundamental property of streaming reasoning: as new events occur, the objects that satisfy a query may change over time. To address this, we propose a new task, Generalized Reasoning Video Segmentation (GRVS), where the target set depends on query-relevant events accumulated over the observed video history. As new evidence accumulates, the target set can evolve from empty to one or multiple objects, requiring a model to continuously maintain target validity while localizing the valid objects currently in view. To evaluate GRVS, we build Isekai, a diagnostic benchmark of 60 videos and 197 queries across 11 temporal query families, with 12,569 scoring points whose target sets are causally annotated over time. We further propose Sekai, an online framework with two complementary components: an Event Memory Reasoner (EMR) that maintains query-relevant events and evolving target states in structured memory, and a Memory-Grounded Decoder (MGD) that identifies valid targets from this memory and localizes them in the current frame. Experiments show that Sekai significantly outperforms existing online RVS methods, achieving 50.33 J&F and improving over the strongest baseline by 3.2 points. Code, example annotations, and evaluation scripts are available at https://github.com/zenastangbi/Sekai.
Figures & tables
| Benchmark | Causal labels | Empty | Multi | Dynamic membership | Evidence age | Temporal operators | Temporal grounding |
|---|---|---|---|---|---|---|---|
| ReasonVOS ( Bai et al., 2024 ) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | Object tracks |
| ReVOS ( Yan et al., 2024 ) | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | Object tracks |
| MeViS v2 ( Ding et al., 2025 ) | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | Object tracks |
| GroundMoRe ( Deng et al., 2025 ) | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | Motion intervals |
| ORVOSB ( Liu et al., 2026 ) | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | Validity intervals |
| Isekai | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Event records |
| Method | J | F | J&F | Now | Ever | Latest | Never | Other | Hist. |
|---|---|---|---|---|---|---|---|---|---|
| OnlineRefer ( Wu et al., 2023 ) | 28.08 | 33.38 | 30.73 | 20.92 | 39.47 | 34.75 | 29.10 | 25.93 | 48.87 |
| SAMWISE ( Cuttano et al., 2025 ) | 33.52 | 37.07 | 35.30 | 22.30 | 46.94 | 40.17 | 34.07 | 27.52 | 57.93 |
| CoT-RVS ( Kao et al., 2026 ) | 46.43 | 47.90 | 47.16 | 59.31 | 39.52 | 25.81 | 18.00 | 53.63 | 22.15 |
| StreamingRVOS ( Zhang et al., 2026b ) | 39.34 | 43.23 | 41.28 | 29.40 | 55.18 | 47.26 | 38.17 | 33.28 | 68.71 |
| Sekai w/o memory | 44.33 | 46.89 | 45.61 | 34.60 | 59.88 | 50.73 | 37.51 | 36.43 | 63.93 |
| Sekai | 49.09 | 51.58 | 50.33 | 42.04 | 64.46 | 44.76 | 27.00 | 48.42 | 65.97 |
| Method / Configuration | Online | ReasonVOS | ReVOS (Overall) |
|---|---|---|---|
| VISA (13B, IT) ( Yan et al., 2024 ) | ✗ | – | 50.90 |
| GLUS (7B) ( Lin et al., 2025 ) | ✗ | 49.90 | 54.90 |
| CoT-RVS (GPT-4o, offline) ( Kao et al., 2026 ) | ✗ | 65.50 | 55.90 |
| Refer-Agent (9B) ( Jiang et al., 2026 ) | ✗ | 69.80 | 61.30 |
| VideoSEG-O3 (4B) ( Dai et al., 2026 ) | ✗ | 62.90 | 67.70 |
| OnlineRefer ( Wu et al., 2023 ; Bai et al., 2024 ) | ✓ | 38.70 | – |
| Module | Condition | All J&F | Hist. J&F |
|---|---|---|---|
| — | Sekai | 50.33 | 65.97 |
| EMR | w/o memory | 45.61 | 63.93 |
| EMR | No state labels | 48.57 | 62.87 |
| MGD | Unadapted locator | 27.66 | 40.34 |
| MGD | SAM2.1 segmentor | 47.92 | 60.26 |
| GT boxes + SAM 3.1 | 87.80 | 90.85 | |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Meaning | Scoring points | Example | |
| Now | Participates now | 60 | 3,821 | “Which dog is rearing up?” |
| Ever | Participated so far | 60 | 3,820 | “Which dogs have reared up so far?” |
| Latest | Participant of the most recently begun occurrence | 18 | 1,151 | “Which dog most recently reared up?” |
| Never | No occurrence confirmed so far | 21 | 1,339 | “Which dog has not reared up?” |
| First | Participant of the earliest-begun occurrence | 5 | 306 | “Which dog reared up first?” |
| Past-not-present | Participated before, but not now | 4 | 246 | “Which dogs have reared up but are not rearing now?” |
| Measure | Value |
|---|---|
| Event annotations (20 audited videos; 36 independent, 34 prior events) | |
| Strictly matched events | 29 |
| Independent-event match rate | 80.6% |
| Prior-event coverage | 85.3% |
| Mean interval IoU among matched events | 0.916 |
| Matched intervals with IoU | 100% |
| Backend | Between anchors | |||
|---|---|---|---|---|
| Filled rectangle | Hold | 28.39 | 20.57 | 24.48 |
| GrabCut | Hold | 30.00 | 27.35 | 28.67 |
| SEEM (FocalT-v0) | Hold | 29.73 | 28.78 | 29.26 |
| SAM2.1 (Hiera-Large) | Propagate | 46.47 | 49.36 | 47.92 |
| GT boxes + SAM 3.1 | Propagate | 87.28 | 88.31 | 87.80 |
| Method | J | F | J&F |
|---|---|---|---|
| CoT-RVS, causal adaptation | 46.43 | 47.90 | 47.16 |
| Sekai | 49.09 | 51.58 | 50.33 |
| Method | In window | 1–8 | 9–24 | 25+ |
|---|---|---|---|---|
| Points | 972 | 1,211 | 2,522 | 2,485 |
| OnlineRefer | 58.08 | 53.47 | 50.37 | 49.99 |
| SAMWISE | 61.38 | 59.35 | 57.95 | 56.78 |
| CoT-RVS | 26.64 | 27.36 | 24.23 | 17.19 |
| StreamingRVOS | 74.61 | 72.79 | 69.27 | 68.19 |
| Sekai w/o memory | 67.11 | 66.99 | 67.86 | 60.14 |
| Method | targets | target | targets |
|---|---|---|---|
| Points | 5,379 | 6,162 | 1,028 |
| OnlineRefer ( Wu et al., 2023 ) | 4.60 | 53.32 | 34.82 |
| SAMWISE ( Cuttano et al., 2025 ) | 4.92 | 59.13 | 44.66 |
| CoT-RVS ( Kao et al., 2026 ) | 75.76 | 21.45 | 30.07 |
| StreamingRVOS ( Zhang et al., 2026b ) | 5.63 | 70.74 | 50.91 |
| Sekai w/o memory | 17.14 | 65.39 | 49.44 |
| Method / Configuration | Online | J&F |
|---|---|---|
| LMPM ( Ding et al., 2023 ) | ✗ | 38.30 |
| DsHmp ( He & Ding, 2024 ) | ✗ | 40.80 |
| LMPM++ ( Ding et al., 2025 ) | ✗ | 43.90 |
| VISA ( Yan et al., 2024 ) (MeViS v2 evaluation) | ✗ | 40.70 |
| Sekai (ours) | ✓ | 50.45 |
Temporal Video Understanding
Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.
See all 384 papersThink Visually Across Space and Time: Grounded Reasoning for Video Segmentation
VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation
[SEG] during the RL stage, we introduce SEG-aware logit calibration, which integrates pixel-wise segmentation feedback directly into the token-level logits. Furthermore, we design a decoupled thinking trace to hierarchically decompose the reasoning process into temporal, spatial, and linguistic dimensions, and construct VTS-CoT, a specialized cold-start dataset featuring comprehensive reasoning trajectories. The code and models will be released at https://github.com/Dmmm1997/VideoSEG-O3.