TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding

Yibo Ma; Qianqian Zhang; Peng Liu; Tiancheng Zhao

cs.CLSep 25, 2026

TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understanding

Authors: Yibo Ma, Qianqian Zhang, Peng Liu, Tiancheng Zhao

Organizations: Om AI Research

Abstract

Streaming video understanding requires models to interpret evidence as it arrives, yet current evaluations often report task scores without specifying when evidence becomes valid, how visual history is maintained, or how responses are triggered. As a result, similar scores may correspond to different workloads, failure modes, and operational behavior. We introduce TRACE (Temporal Audit and Condition-aware Evaluation), a condition-aware benchmark and evaluation framework that makes these factors explicit. TRACE combines temporally audited visual tasks with evidence timing and instruction-dependent trigger annotations, a unified causal Core--Adapter protocol that controls information availability while recording actual history processing and response events, and multidimensional reporting of answer quality, timeliness, response-selection behavior, workload, completion, and reliability. On 1,240 records from 517 videos, we evaluate eight publicly available models or systems in eight configurations. We find that nearly identical QA accuracy can mask substantial differences in completion, answer validity, and generation workload, while proactive performance separates into response quality, response delay, false alarms (responses emitted while no target window is currently valid and a later one remains), and missed target windows. These results show that streaming-video performance should be interpreted as execution-conditioned system behavior rather than a single score. Our benchmark and code can be accessed at https://github.com/om-ai-lab/trace-bench.

Figures & tables

Benchmark	Vid.	Rec.	QA	Pro- active	Trigger rules	Evidence timing	Audited ground truth	Accu- racy	Latency (QA)	Invalid outputs	Accu- racy	Response timing	FA / Miss	Work- load	Comple- tion
			Task		Annotation			QA metrics			Proactive metrics			Run-level
StreamingBench [ 12 ]	900	4,500	✓	✓	$\times$	$\times$	$\times$	✓	$\times$	$\times$	✓	$\times$	$\times$	$\times$	$\times$
OVO-Bench [ 9 ]	644	$\sim$ 3.1k	✓	✓	$\times$	$\times$	$\times$	✓	$\times$	$\times$	✓	✓	$\times$	$\times$	$\times$
OVBench [ 6 ]	1,463	4,874	✓	$\times$	$\times$	$\times$	$\times$	✓	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
RIVER [ 18 ]	1,067	4,278	✓	✓	$\times$	✓	$\times$	✓	$\times$	$\times$	✓	✓	$\times$	$\times$	$\times$
ProactiveVideoQA [ 23 ]	1,377	1,427	✓	✓	$\times$	$\times$	$\times$	✓	$\times$	$\times$	✓	✓	$\times$	$\times$	$\times$

Table 1: Comparison with released streaming-video benchmarks most directly related to TRACE’s scope. All listed projects provide public data and evaluation code with causal streaming access, verified in our 2026-07 release audit; a cross marks absence from the released definitions we audited, not a deficiency of the underlying work.

Figure 1: TRACE overview from causal video input to condition-aware measurement. The top row connects the controlled Core, model-specific Adapter, evaluated model or system, and scoring layer. The middle timeline separates QA video-time evidence availability from runtime response events; the bottom timeline separates proactive trigger validity, response onset, and response content. These observation points support the quality, timing, extra-output, workload, completion, and reliability measurements reported by TRACE. Timelines are schematic.

Figure 2: Execution conditions recorded by TRACE. Native Streaming updates persistent state as frames arrive, whereas Non-native Streaming reconstructs a legal causal prefix or window at query time. Model + Adapter and complete-system boundaries specify which components contribute to reported measurements; deployment is declared separately. Proactive configurations in the standard comparison receive one monitoring instruction and self-initiate their responses. These properties define measurement conditions, not capability levels.

Task / dimension	Metric	Definition
QA quality	Accuracy $\uparrow$	Correct, uniquely parsed records divided by all records
QA reliability	Completion Rate $\uparrow$	Successfully completed records divided by all QA records
QA response	Response Latency $\downarrow$	Question arrival $r_{q}$ to completion receipt $r_{\mathrm{end}}$
QA response	Time to First Token (TTFT) $\downarrow$	Question arrival $r_{q}$ to first output token $r_{1}$
QA workload	Submitted Image Count $\downarrow$	Actual image submissions, including repeats
QA workload	Total Output Tokens $\downarrow$	All tokens generated during the QA run; Section 5 reports the portion observed through interface telemetry

Table 2: Main evaluation metrics. Arrows indicate preferred directions when task population, completion conditions, and measurement boundaries are comparable.

Model/system	State and input organization	Proactive trigger	Deployment and evaluation boundary
AURA	QA: Non-native prefix processing at query time; Proactive: persistent history with incremental visual-state updates	autonomous	Local service; model + Adapter
MOSS-VL	Native; real-time session and asynchronous frame queue	autonomous	Local weights; model + Adapter
MOSS-Preview	Native; real-time session and asynchronous frame queue	autonomous	Local weights; model + Adapter
LiveCC	Native; persistent session and streaming generation	autonomous	Local weights; model + Adapter
ThinkStream	Native; two-frame blocks and persistent state	autonomous	Local weights; model + Adapter
VideoLLM-Online	Native; per-frame visual representations and persistent key-value (KV) state	autonomous	Local weights; model + Adapter

Table 3: Per-model configurations and result boundaries.

Figure 3: QA accuracy alongside median response latency (left) and recorded output tokens per scheduled QA record (right, logarithmic axis). Shapes identify native model + Adapter, complete-system, non-native prefix-input, and native duplex configurations. In both panels, higher accuracy together with lower latency or lower generation volume appears toward the upper left. The axes report observed measurements under the boundaries in Table 3 , not hardware-normalized efficiency.

(a) Autonomous model + Adapter
			Response behavior		Workload
Model	In-window Acc	Median delay (s)	False- alarm	Miss rate	Submitted images	Output tokens
LiveCC	12.98%	1.08	65.0%	0.31%	23,243	78,478
AURA	7.92%	1.15	59.1%	47.64%	439,400	50,957
MOSS-VL	8.05%	0.33	41.1%	47.24%	34,520	74,672
MOSS-Preview	4.45%	0.20	77.7%	35.67%	34,520	193,517

Table 5: Proactive results by response track. In-window Accuracy and Miss are window-level; False-alarm Rate is a global response-episode ratio; Median Response Delay is conditional on answered windows with an observed onset. Delay coverage is reported in Appendix B.2 . Image and token columns are observed run totals over the scheduled population and follow the accounting boundary of Table 4 .

Figure 4: Proactive Response results on the standard set. (a) In-window Accuracy with video-clustered 95% confidence intervals versus Median Response Delay. Higher accuracy and lower delay appear toward the upper left. (b) Response-selection behavior, summarizing two complementary forms of response-selection error; lower values on both axes are preferred. Marker shapes distinguish native model + Adapter, complete-system, and native duplex diagnostic configurations.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: In-window Accuracy by Proactive trigger type for seven standard response tracks. Bars are current-v4 point estimates; legend entries give target-window counts. Shading separates the standard model-side and complete-system (JoyAI) tracks. Trigger subsets differ in size and content and are descriptive rather than controlled difficulty groups.

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 132 papers

StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding
Mar 23, 2026Guowei Tang, Tianwen Qian, Huanran Zheng +2Streaming Video Understanding Video-Language Model Evaluation
East China Normal University, Shanghai, China
PACE: Plan, Attend, Check, and Emit for Training-Free Response Timing in Streaming Video Understanding
Oct 20, 2026Siqi Guo, Yuxia Chen, Wenbo Zhu +4Streaming Video Understanding Video QA
Nanyang Technological University · University of Electronic Science and Technology of China · University of California, Berkeley +3
TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understanding
Aug 23, 2026Pengyiang Liu, Junbo Niu, Xiaoyang Hu +4Video QA Long-Video Understanding
Colab, Beihang University · Shanghai AI Laboratory · Peking University

Mar 23, 2026J/K move · Enter open · S save

StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding

Guowei Tang, Tianwen Qian, Huanran Zheng, Yifei Wang+1

East China Normal University, Shanghai, China

Streaming Video Understanding Video-Language Model Evaluation Efficient VLM Inference

Open paper AskPDF arXiv

Real-time, continuous understanding of visual signals is essential for real-world interactive AI applications, and poses a fundamental system-level challenge. Existing research on streaming video understanding, however, typically focuses on isolated aspects such as question-answering accuracy under limited visual context or improvements in encoding efficiency, while largely overlooking practical deployability under realistic resource constraints. To bridge this gap, we introduce StreamingEval, a unified evaluation framework for assessing the streaming video understanding capabilities of Video-LLMs under realistic constraints. StreamingEval benchmarks both mainstream offline models and recent online video models under a standardized protocol, explicitly characterizing the trade-off between efficiency, storage and accuracy. Specifically, we adopt a fixed-capacity memory bank to normalize accessible historical visual context, and jointly evaluate visual encoding efficiency, text decoding latency, and task performance to quantify overall system deployability. Extensive experiments across multiple datasets reveal substantial gaps between current Video-LLMs and the requirements of realistic streaming applications, providing a systematic basis for future research in this direction. Codes will be released at https://github.com/wwgTang-111/StreamingEval1.

Abstract

Streaming Video Understanding

Explore similar work

StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding

PACE: Plan, Attend, Check, and Emit for Training-Free Response Timing in Streaming Video Understanding

TRACE: Temporal Retrieval with Anchored and Convergent Evidence for Long-Horizon Video Understanding