Supervising the Visible State: Separating Representation from View Selection in Streaming Video
Organizations: Westlake University · USTC · ZJU · SMU ·
Abstract
An action seen once and the same action seen three times receive the same action-recognition label, so that supervision does not require a model to count. Answers shared across histories that differ in event order, accumulation, or relations leave those distinctions unconstrained; we call this state under-specification. Visible-State Supervision (ViSS) addresses it by training on source-grounded probes of event identity, order, accumulation, relation change, and answer sufficiency; under the same recent-frame interface it improves OVO-Bench over a strict-matched generic temporal supervision baseline while remaining comparable on StreamingBench, with the largest gain in hallucination detection. Test-Time Causal View Search (TVS) then adapts evidence access while keeping the learned model fixed, expanding from recent frames to sparse event and compact history views on demand. The two interventions separate representation from view selection: supervision particularly benefits hallucination detection and causal reasoning, access particularly benefits event recall and forward reasoning. Together they reach the best reported accuracy among the published systems compared here on StreamingBench and OVO-Bench at a fraction of the processed frames of exhaustive selection, and strict replay shows that most of the remaining gap to a three-view oracle is a correct answer that was later replaced. Streaming accuracy depends on what supervision asks the model to keep, what evidence inference exposes, and whether the model keeps a correct answer once it has one.
Figures & tables
| Real-Time Visual Perception | Backward Tracing | Forward Active Responding | |||||||||||||||
| Model | Frames | OCR | ACR | ATR | STU | FPD | OJR | Avg. | EPM | ASI | HLD | Avg. | REC | SSR | CRR | Avg. | Overall |
| Human | – | 93.96 | 92.57 | 94.83 | 92.70 | 91.09 | 94.02 | 93.20 | 92.59 | 93.02 | 91.37 | 92.33 | 95.48 | 89.67 | 93.56 | 92.90 | 92.81 |
| Proprietary MLLMs | |||||||||||||||||
| Gemini 1.5 Pro | 1 fps | 85.91 | 66.97 | 79.31 | 58.43 | 63.37 | 61.96 | 69.32 | 58.59 | 76.35 | 52.69 | 62.54 | 35.53 | 74.24 | 61.67 | 57.15 | 63.00 |
| GPT-4o | 64 | 69.80 | 64.22 | 71.55 | 51.12 | 70.30 | 59.78 | 64.46 | 57.91 | 75.68 | 48.66 | 60.75 | 27.58 | 73.21 | 59.40 | 53.40 | 59.54 |
| Open-source offline MLLMs | |||||||||||||||||
| Model | Frames | OP | CR | CS | ATP | EU | TR | PR | SU | ACP | CT | ALL |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Human | – | 89.47 | 92.00 | 93.60 | 91.47 | 95.65 | 92.52 | 88.00 | 88.75 | 89.74 | 91.30 | 91.46 |
| Proprietary MLLMs | ||||||||||||
| Gemini 1.5 Pro | 1 fps | 79.02 | 80.47 | 83.54 | 79.67 | 80.00 | 84.74 | 77.78 | 64.23 | 71.95 | 48.70 | 75.69 |
| GPT-4o | 64 | 77.11 | 80.47 | 83.91 | 76.47 | 70.19 | 83.80 | 66.67 | 62.19 | 69.12 | 49.22 | 73.28 |
| Claude 3.5 Sonnet | 20 | 80.49 | 77.34 | 82.02 | 81.73 | 72.33 | 75.39 | 61.11 | 61.79 | 69.32 | 43.09 | 72.44 |
| Open-source offline MLLMs | ||||||||||||
| Supervision | SB | OVO |
|---|---|---|
| Base | 80.52 | 58.97 |
| Strict-matched generic | 84.97 | 63.48 |
| ViSS w/o sufficiency | 84.89 | 63.52 |
| ViSS | 84.77 | 65.21 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| View | Recent support | Historical support | Frames |
|---|---|---|---|
| recent | 4 frames from the latest 4 seconds | none | 4 |
| event | the same recent 4 | up to 16 ranked frames, standard resolution | 20 |
| compact history | 8 frames from the latest 8 seconds | up to 24 ranked frames, reduced resolution | 32 |
| Component | Setting |
|---|---|
| Historical ranking | RGB descriptors; novelty, trajectory curvature, and recency with weights |
| Candidate construction | up to 64 prefix-grid positions, plus reserved recent IDs after filtering |
| Calibration | 300 queries from 300 source-video-disjoint nonbenchmark videos |
| Acquisition threshold | |
| Relative-margin tolerance | |
| Inference budget | at most 3 calls; at most 32 frames in any one call |
| Family | Visible-state target | Rows |
|---|---|---|
| source-aligned anchors | general causal video understanding and capability retention | 3,594 |
| repetition-counting probes | accumulated event count over an ordered visual prefix | 2,172 |
| prefix-32 state probes | event state, temporal order, and accumulation over longer prefixes | 1,012 |
| sufficiency anchors | whether the available causal evidence supports an answer | 116 |
| capability probes | backward state retrieval, action sequence, and counting | 783 |
| Total | 7,677 |
| Source and input | Supervision question | Direct target |
|---|---|---|
| RepCount annotations; 16 frames | The frames are chronological causal observations. Count how many complete situp repetitions have fully finished by this point. An incomplete motion does not count. Reply with one integer only. | 2 |
| Teacher capability probe; 12 frames | What gesture did the woman make with her right hand before she began speaking with her hand open and palm facing up? | D : She pointed her index finger upwards. |
| Setting | Value |
|---|---|
| Model and input | |
| Initialization | Qwen3-VL-8B-Instruct; fresh run from public weights |
| Trainable modules | language model and multimodal projector; vision tower frozen |
| Input limits | 8,192 tokens; 200,704 pixels per image |
| Loss | assistant targets only; serialized example weights |
| Data and optimization | |
| Training attribute | Strict-matched generic | ViSS |
|---|---|---|
| Rows | 7,677 | 7,677 |
| Visual inputs and exposure | identical | identical |
| Weights, stages, and order | identical | identical |
| Optimization, initialization, seed | identical | identical |
| Question–target formulation | generic temporal QA | visible-state probes |
| StreamingBench | 84.97 | 84.77 |
| Seed pair | StreamingBench | OVO-Bench | HLD |
|---|---|---|---|
| 1 | 2,120 / 2,115 | 63.48 / 65.21 | 125 / 140 |
| 2 | 2,117 / 2,113 | 63.21 / 64.90 | 124 / 138 |
| 3 | 2,116 / 2,112 | 63.63 / 65.37 | 126 / 140 |
| Mean score | 84.88 / 84.70 | 63.44 / 65.16 | 67.20 / 74.91 |
| Mean paired difference |
| Configuration | Rows | SB | OVO | CT | REC | HLD |
|---|---|---|---|---|---|---|
| full ViSS | 7,677 | 84.77 | 65.21 | 44.04 | 21.20 | 75.27 |
| without sufficiency | 7,561 | 84.89 | 63.52 | 46.63 | 20.63 | 70.97 |
| StreamingBench | OVO-Bench | |||||
| Access policy | Acc. | Calls/q | Frames/q | Macro | Calls/q | Frames/q |
| Fixed one-view controls | ||||||
| recent only | 84.77 | 1.00 | 4.00 | 65.21 | 1.00 | 4.00 |
| event only | 85.69 | 1.00 | 19.51 | 67.67 | 1.00 | 19.79 |
| history only | 86.45 | 1.00 | 29.97 | 68.84 | 1.00 | 30.54 |
| static history-17 | 85.69 | 1.00 | 17.00 | – | – | – |
| Group | Task | Base | ViSS | Oracle | |
|---|---|---|---|---|---|
| RT | Optical character recognition (OCR) | 95.30 | 95.30 | 96.64 | 97.32 |
| RT | Attribute recognition (ATR) | 82.76 | 84.48 | 82.76 | 87.93 |
| RT | Object recognition (OJR) | 82.61 | 83.15 | 79.89 | 87.50 |
| RT | Action recognition (ACR) | 80.73 | 82.57 | 80.73 | 87.16 |
| RT | Future prediction (FPD) | 73.27 | 75.25 | 80.20 | 83.17 |
| RT | Spatial understanding (STU) | 73.60 | 72.47 | 67.42 | 78.09 |
| Diagnostic | Interpretation | StreamingBench | OVO-Bench |
|---|---|---|---|
| View recoverability | A broader causal view contains a correct proposal | 43.7% (166/380) | 37.1% (447/1,206) |
| Policy recovery | TVS returns the correct answer when one is available | 63.3% (105/166) | 71.1% (318/447) |
| Recent-answer preservation | TVS retains an initially correct recent proposal | 96.9% (2,049/2,115) | 92.3% (1,688/1,829) |
| Failure mechanism | Trace-level criterion | StreamingBench | OVO-Bench |
| Acquisition miss | No correct proposal is computed along the deployed path | 32 (25.2%) | 53 (19.6%) |
| Acceptance miss | A correct proposal is computed but never retained | 10 (7.9%) | 23 (8.5%) |
| Replacement error | A retained correct proposal is subsequently replaced | 85 (66.9%) | 194 (71.9%) |
| Total | Errors containing at least one correct replayed proposal | 127 | 270 |
| Acquired view | Frames | Proposal | Margin | Retained |
|---|---|---|---|---|
| recent | 4 | D | 0.5084 | D |
| event | 20 | B | 0.1228 | B |
| compact history | 29 | C | 0.2416 | C |
Streaming Video Understanding
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.
See all 132 papers