cs.CLSep 25, 2026

TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding

Authors: Yibo Ma, Qianqian Zhang, Peng Liu, Tiancheng Zhao

Organizations: Om AI Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Streaming video understanding requires models to interpret evidence as it arrives, yet current evaluations often report task scores without specifying when evidence becomes valid, how visual history is maintained, or how responses are triggered. As a result, similar scores may correspond to different workloads, failure modes, and operational behavior. We introduce TRACE (Temporal Audit and Condition-aware Evaluation), a condition-aware benchmark and evaluation framework that makes these factors explicit. TRACE combines temporally audited visual tasks with evidence timing and instruction-dependent trigger annotations, a unified causal Core--Adapter protocol that controls information availability while recording actual history processing and response events, and multidimensional reporting of answer quality, timeliness, response-selection behavior, workload, completion, and reliability. On 1,240 records from 517 videos, we evaluate eight publicly available models or systems in eight configurations. We find that nearly identical QA accuracy can mask substantial differences in completion, answer validity, and generation workload, while proactive performance separates into response quality, response delay, false alarms (responses emitted while no target window is currently valid and a later one remains), and missed target windows. These results show that streaming-video performance should be interpreted as execution-conditioned system behavior rather than a single score. Our benchmark and code can be accessed at https://github.com/om-ai-lab/trace-bench.

Figures & tables

   TaskAnnotationQA metricsProactive metricsRun-level
BenchmarkVid.Rec.QAPro- activeTrigger rulesEvidence timingAudited ground truthAccu- racyLatency (QA)Invalid outputsAccu- racyResponse timingFA / MissWork- loadComple- tion
StreamingBench [ 12 ]9004,500✓✓×\times×\times×\times✓×\times×\times✓×\times×\times×\times×\times
OVO-Bench [ 9 ]644∼\sim 3.1k✓✓×\times×\times×\times✓×\times×\times✓✓×\times×\times×\times
OVBench [ 6 ]1,4634,874✓×\times×\times×\times×\times✓×\times×\times×\times×\times×\times×\times×\times
RIVER [ 18 ]1,0674,278✓✓×\times✓×\times✓×\times×\times✓✓×\times×\times×\times
ProactiveVideoQA [ 23 ]1,3771,427✓✓×\times×\times×\times✓×\times×\times✓✓×\times×\times×\times
Table 1: Comparison with released streaming-video benchmarks most directly related to TRACE’s scope. All listed projects provide public data and evaluation code with causal streaming access, verified in our 2026-07 release audit; a cross marks absence from the released definitions we audited, not a deficiency of the underlying work.
Figure 1: TRACE overview from causal video input to condition-aware measurement. The top row connects the controlled Core, model-specific Adapter, evaluated model or system, and scoring layer. The middle timeline separates QA video-time evidence availability from runtime response events; the bottom timeline separates proactive trigger validity, response onset, and response content. These observation points support the quality, timing, extra-output, workload, completion, and reliability measurements reported by TRACE. Timelines are schematic.
Figure 2: Execution conditions recorded by TRACE. Native Streaming updates persistent state as frames arrive, whereas Non-native Streaming reconstructs a legal causal prefix or window at query time. Model + Adapter and complete-system boundaries specify which components contribute to reported measurements; deployment is declared separately. Proactive configurations in the standard comparison receive one monitoring instruction and self-initiate their responses. These properties define measurement conditions, not capability levels.
Task / dimensionMetricDefinition
QA qualityAccuracy ↑\uparrowCorrect, uniquely parsed records divided by all records
QA reliabilityCompletion Rate ↑\uparrowSuccessfully completed records divided by all QA records
QA responseResponse Latency ↓\downarrowQuestion arrival rqr_{q} to completion receipt rendr_{\mathrm{end}}
QA responseTime to First Token (TTFT) ↓\downarrowQuestion arrival rqr_{q} to first output token r1r_{1}
QA workloadSubmitted Image Count ↓\downarrowActual image submissions, including repeats
QA workloadTotal Output Tokens ↓\downarrowAll tokens generated during the QA run; Section 5 reports the portion observed through interface telemetry
Table 2: Main evaluation metrics. Arrows indicate preferred directions when task population, completion conditions, and measurement boundaries are comparable.
Model/systemState and input organizationProactive triggerDeployment and evaluation boundary
AURAQA: Non-native prefix processing at query time; Proactive: persistent history with incremental visual-state updatesautonomousLocal service; model + Adapter
MOSS-VLNative; real-time session and asynchronous frame queueautonomousLocal weights; model + Adapter
MOSS-PreviewNative; real-time session and asynchronous frame queueautonomousLocal weights; model + Adapter
LiveCCNative; persistent session and streaming generationautonomousLocal weights; model + Adapter
ThinkStreamNative; two-frame blocks and persistent stateautonomousLocal weights; model + Adapter
VideoLLM-OnlineNative; per-frame visual representations and persistent key-value (KV) stateautonomousLocal weights; model + Adapter
Table 3: Per-model configurations and result boundaries.
Figure 3: QA accuracy alongside median response latency (left) and recorded output tokens per scheduled QA record (right, logarithmic axis). Shapes identify native model + Adapter, complete-system, non-native prefix-input, and native duplex configurations. In both panels, higher accuracy together with lower latency or lower generation volume appears toward the upper left. The axes report observed measurements under the boundaries in Table 3 , not hardware-normalized efficiency.
   Response behaviorWorkload
ModelIn-window AccMedian delay (s)False- alarmMiss rateSubmitted imagesOutput tokens
(a) Autonomous model + Adapter
LiveCC12.98%1.0865.0%0.31%23,24378,478
AURA7.92%1.1559.1%47.64%439,40050,957
MOSS-VL8.05%0.3341.1%47.24%34,52074,672
MOSS-Preview4.45%0.2077.7%35.67%34,520193,517
Table 5: Proactive results by response track. In-window Accuracy and Miss are window-level; False-alarm Rate is a global response-episode ratio; Median Response Delay is conditional on answered windows with an observed onset. Delay coverage is reported in Appendix B.2 . Image and token columns are observed run totals over the scheduled population and follow the accounting boundary of Table 4 .
Figure 4: Proactive Response results on the standard set. (a) In-window Accuracy with video-clustered 95% confidence intervals versus Median Response Delay. Higher accuracy and lower delay appear toward the upper left. (b) Response-selection behavior, summarizing two complementary forms of response-selection error; lower values on both axes are preferred. Marker shapes distinguish native model + Adapter, complete-system, and native duplex diagnostic configurations.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: In-window Accuracy by Proactive trigger type for seven standard response tracks. Bars are current-v4 point estimates; legend entries give target-window counts. Shading separates the standard model-side and complete-system (JoyAI) tracks. Trigger subsets differ in size and content and are descriptive rather than controlled difficulty groups.
More in this topic

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 132 papers

Explore similar work

CardsList
Mar 23, 2026

StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding

Real-time, continuous understanding of visual signals is essential for real-world interactive AI applications, and poses a fundamental system-level challenge. Existing research on streaming video understanding, however, typically focuses on isolated aspects such as question-answering accuracy under limited visual context or improvements in encoding efficiency, while largely overlooking practical deployability under realistic resource constraints. To bridge this gap, we introduce StreamingEval, a unified evaluation framework for assessing the streaming video understanding capabilities of Video-LLMs under realistic constraints. StreamingEval benchmarks both mainstream offline models and recent online video models under a standardized protocol, explicitly characterizing the trade-off between efficiency, storage and accuracy. Specifically, we adopt a fixed-capacity memory bank to normalize accessible historical visual context, and jointly evaluate visual encoding efficiency, text decoding latency, and task performance to quantify overall system deployability. Extensive experiments across multiple datasets reveal substantial gaps between current Video-LLMs and the requirements of realistic streaming applications, providing a systematic basis for future research in this direction. Codes will be released at https://github.com/wwgTang-111/StreamingEval1.

Guowei Tang, Tianwen Qian, Huanran Zheng +2

East China Normal University, Shanghai, China

Streaming Video UnderstandingVideo-Language Model EvaluationEfficient VLM Inference
AskPDF
Oct 20, 2026

PACE: Plan, Attend, Check, and Emit for Training-Free Response Timing in Streaming Video Understanding

Streaming video understanding requires a model to decide not only what to answer but also when to answer. Evidence arrives progressively, and recent benchmarks identify premature responses, issued before the queried event has concluded, as a recurring failure mode. Existing approaches often learn response timing through additional training, while training-free alternatives typically rely on scalar gating signals that do not explicitly represent when an answer becomes final. We organize chunk-wise streaming errors into three patterns: sampling variation can make an answer unstable, a stable answer may lack support in the observed video, or an answer may be supported by the current evidence while the event remains unfinished. We propose PACE (Plan, Attend, Check, and Emit), a training-free framework designed around these three failure modes. Given a question, PACE drafts a plan specifying what to observe and what will signal completion. As the stream unfolds, it invokes an off-the-shelf video LLM at informative moments, evaluates each draft for stability, grounding in its cited frames, and completion, and records verified observations in an event ledger. Experiments on ProactiveVideoQA and OVO-Bench Forward show that PACE improves over the official protocols by up to 5.9 average PAUC points and 6.2 average accuracy points across proprietary and open-source backbones, yielding state-of-the-art performance.

Siqi Guo, Yuxia Chen, Wenbo Zhu +4

Nanyang Technological University · University of Electronic Science and Technology of China · University of California, Berkeley +3

Streaming Video UnderstandingVideo QA
AskPDF
Aug 23, 2026

TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understanding

A long-video answer is evidence-supported only when the frames decoded from the video cover every event the answer depends on. Existing evaluations score final-answer correctness or predicted evidence intervals, but the frames a method decodes before answering are rarely audited, so correct answers can still rest on incomplete observation. We introduce VES-Bench, a 600-question benchmark of Temporal Ordering and Event Counting items over 348 public long videos. Each item carries a jointly necessary set of evidence intervals, letting us audit at three strictness levels whether a method's decoded frames cover every one of them. We also propose TRACE, a training-free agent that grounds answers in raw visual clips, builds an evidence bundle round by round, and stops only when the answer stabilises as the bundle grows and a final pass over the same clips returns the same answer. Under a same-backbone audit, TRACE answers 50.7% of questions correctly with at least two decoded frames inside every evidence interval, at 98.7 frames per question: over 10 points above uniform decoding at 128 frames (40.2%), and within 2.6 points of uniform decoding at 256 frames at 0.39x its frame cost, while reaching the highest answer accuracy in the audit (63.5%). TRACE also stays competitive on Video-MME (86.1), LVBench (75.6), and LongVideoBench (75.1).

Pengyiang Liu, Junbo Niu, Xiaoyang Hu +4

Colab, Beihang University · Shanghai AI Laboratory · Peking University

Video QALong-Video UnderstandingLong-Video QAQuery-Aware Video Frame SelectionTemporal Video Grounding
AskPDF