cs.CVSep 27, 2026

OPERA: A Unified Omnimodal Progressive Spatio-Temporal Reasoning Agent for Referring Video Segmentation

Authors: Jingchen Ni, Yuji Wang, Shannan Yan, Haoru Li, Sitong Chen, Chun Yuan

Organizations: Tsinghua University · University of California San Diego · ETH Zurich

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Referring video segmentation with heterogeneous multimodal queries---spanning text, audio, and reference images---demands both robust cross-modal understanding and precise spatio-temporal reasoning. We propose OPERA (Omnimodal Progressive spatio-tEmporal Reasoning Agent), a unified reasoning agent built on a single MLLM that performs dual-axis progressive reasoning via three specialized stages. Along the temporal axis, a Temporal Reasoning Agent narrows the frame search space through coarse-to-fine filtering to identify the most informative key frame. Along the spatial axis, a Distillation Agent establishes what to locate via cross-modal semantic distillation, and a Grounding Agent enhanced with GRPO determines where the target appears, with dense mask propagation completing the pixel-level output. OPERA sets a new state of the art on OmniAVS and Ref-AVS and transfers zero-shot to standard referring video segmentation benchmarks.

Figures & tables

Figure 1: Comparison of Omni-R1 (a) and OPERA (b) on an omnimodal query. Omni-R1 selects a keyframe and localizes the target in two sequential LLM calls without progressive filtering, so an occluded or motion-blurred frame yields mis-localization that propagates into erroneous masks. OPERA instead reasons progressively along two axes: the temporal axis narrows to the most informative keyframe by coarse-to-fine filtering (Temporal Agent), while the spatial axis fixes what to locate (Distillation Agent) and where it appears (Grounding Agent).
Figure 2: Overview of OPERA using a text+sound+image query as an example. Along the spatial axis , the Distillation Agent (§ 3.2) converts non-textual cues into concise textual hints, and the Grounding Agent (§ 3.4) localizes the target on the selected key frame via chain-of-thought reasoning optimized with GRPO. Along the temporal axis , the Temporal Reasoning Agent (§ 3.3) progressively narrows the candidate frame set through uniform sampling, vision-language coarse filtering, and LLM-based semantic reranking to select a single key frame. Predicted bounding boxes are then passed to the Mask Generator for dense pixel-level propagation across all frames.
Figure 3: Sensitivity of the temporal candidate funnel on OmniAVS. (a) Uniform sample count KK : J&F\mathcal{J}\&\mathcal{F} peaks at K=16K{=}16 , degrading both when candidates are too few and when excessive sampling adds near-duplicates. (b) Coarse-filter shortlist size NcN_{c} : Nc=4N_{c}{=}4 is best, with larger shortlists making LLM reranking less reliable.
Figure 4: Qualitative results of OPERA on four examples spanning diverse query modalities. Each row shows video frames with propagated segmentation masks (orange), the selected key frame, and the chain-of-thought grounding trace with predicted bounding box.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: GRPO training reward curves for δ∈{0.3,0.5,0.7}\delta\in\{0.3,0.5,0.7\} over 4,000 steps (solid: raw; dashed: EMA). δ=0.5\delta{=}0.5 rises steadily to 0.80; δ=0.3\delta{=}0.3 reaches ≈0.75{\approx}0.75 but decouples from localization precision; δ=0.7\delta{=}0.7 stays near 0.20 for the first 1,500 steps due to sparse rewards.

Explore similar work

CardsList

  1. ReflexTrack: A Feedback-Driven Agent for Training-Free Referring Video Object Segmentation

    Jul 27, 2026Yuanjia Li, Tianyang Xu, Tao Zhou +3Video Object SegmentationSpatial Grounding

    School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China · Centre for Vision, Speech and Signal Processing, University of Surrey, Guildford, GU2 7XH, UK

  2. SteerSeg: Attention Steering for Reasoning Video Segmentation

    May 14, 2026Ali Cheraghian, Hamidreza Dastmalchi, Abdelwahed Khamis +3Video Object SegmentationVision-Language Model Grounding

    1Macquarie University · 2York University · 3CSIRO Data61 +1

  3. Event-Aware Instructed Assistant for Referring Video Segmentation

    Jun 25, 2026Jinyu Liu, Henghui Ding, Shuting He +1Video Object SegmentationHierarchical

    Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Shanghai University of Finance and Economics, China · Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China