cs.CVMar 23, 2026
SaveColor When It Counts: Grayscale-Guided Online Triggering for Always-On Streaming Video Sensing
Organizations: Queen Mary University of London · Independent Researcher · Durham University · Imperial College London ·
Abstract
Always-on sensing is essential for next-generation edge/wearable AI systems, yet continuous high-fidelity RGB video capture remains prohibitively expensive for resource-constrained mobile and edge platforms. We present a new paradigm for efficient streaming video understanding: grayscale-always, color-on-demand. Through preliminary studies, we discover that color is not always necessary. Sparse RGB frames suffice for comparable performance when temporal structure is preserved via continuous grayscale streams. Building on this insight, we propose ColorTrigger, an online training-free trigger that selectively activates color capture based on windowed grayscale affinity analysis. Designed for real-time edge deployment, ColorTrigger uses lightweight quadratic programming to detect chromatic redundancy causally, coupled with credit-budgeted control and dynamic token routing to jointly reduce sensing and inference costs. On streaming video understanding benchmarks, ColorTrigger achieves 91.6% of full-color baseline performance while using only 8.1% RGB frames, demonstrating substantial color redundancy in natural videos and enabling practical always-on video sensing on resource-constrained devices.
Figures & tables
Figure 1: Towards always-on sensing. (a) Realizing always-on sensing is difficult in practice: the Hi-res RGB video pipeline (such as sensor, ISP, encoding, and wireless transmission) quickly exhausts power on the edge AI system, so devices such as smart glasses typically sustain only about 30–60 minutes of continuous recording [ 35 ] , far from the all-day operation needed for a standby assistant. (b) We propose ColorTrigger, a grayscale-always, color-on-demand paradigm that uses a low-power gray camera as an always-on monitor and sparsely triggers an RGB camera only when needed, enabling always-on video sensing on edge devices.
Figure 2: Preliminary Studies: Color is not always necessary. (a) We uniformly insert RGB frames of the same resolution and quality into an otherwise grayscale video stream and evaluate on StreamingBench (All tasks) using Qwen2.5-VL-7B [ 2 ] . Results show that only a small fraction of RGB frames is sufficient to achieve comparable performance. Furthermore, (b) We extract CLS features from 30 consecutive frames of a 1 fps video using CLIP ViT-B/16 [ 33 ] , revealing redundancy in adjacent frames.
Figure 3: Overview of ColorTrigger. Our framework operates in two stages: (i) a causal online trigger analyzes grayscale features within a sliding window by aggregating them into an affinity matrix that captures temporal redundancy and change and applying a lightweight quadratic program under a credit-based budget; (ii) a dynamic token router adaptively allocates decoder capacity based on the trigger decision, using high-compression tokens for grayscale frames and high-capacity tokens for RGB frames, while preserving temporal order.
| Model | #Frames | RGB (%) | OP | CR | CS | ATP | EU | TR | PR | SU | ACP | CT | All |
| Human | - | - | 89.47 | 92.00 | 93.60 | 91.47 | 95.65 | 92.52 | 88.00 | 88.75 | 89.74 | 91.30 | 91.46 |
| Proprietary MLLMs | |||||||||||||
| Gemini 1.5 Pro [ 42 ] | 1 fps | 100 | 79.02 | 80.47 | 83.54 | 79.67 | 80.00 | 84.74 | 77.78 | 64.23 | 71.95 | 48.70 | 75.69 |
| GPT-4o [ 18 ] | 64 | 100 | 77.11 | 80.47 | 83.91 | 76.47 | 70.19 | 83.80 | 66.67 | 62.19 | 69.12 | 49.22 | 73.28 |
| Claude 3.5 Sonnet [ 1 ] | 20 | 100 | 80.49 | 77.34 | 82.02 | 81.73 | 72.33 | 75.39 | 61.11 | 61.79 | 69.32 | 43.09 | 72.44 |
| Open-Source Video MLLMs | |||||||||||||
Table 1: Performance comparison on StreamingBench [ 23 ] focusing on Real-Time Visual Understanding tasks. Real-Time Visual Understanding encompasses Object Perception (OP), Causal Reasoning (CR), Clips Summarization (CS), Attribute Perception (ATP), Event Understanding (EU), Text-Rich Understanding (TR), Prospective Reasoning (PR), Spatial Understanding (SU), Action Perception (ACP), and Counting (CT). “RGB(%)” represents the percentage of color frames remaining after triggering, where indicates full color, means all grayscale, and signifies a 91.9% reduction in color frames.
| Model | #Frames | RGB (%) | Real-Time Visual Perception | Backward Tracing | Forward Active Responding | Overall | ||||||||||||
| OCR | ACR | ATR | STU | FPD | OJR | Avg. | EPM | ASI | HLD | Avg. | REC | SSR | CRR | Avg. | Avg. | |||
| Human Agents | - | - | 94.0 | 92.6 | 94.8 | 92.7 | 91.1 | 94.0 | 93.2 | 92.6 | 93.0 | 91.4 | 92.3 | 95.5 | 89.7 | 93.6 | 92.9 | 92.8 |
| Proprietary MLLMs | ||||||||||||||||||
| Gemini 1.5 Pro [ 42 ] | 1 fps | 100 | 87.3 | 67.0 | 80.2 | 54.5 | 68.3 | 67.4 | 70.8 | 68.6 | 75.7 | 52.7 | 62.3 | 35.5 | 74.2 | 61.7 | 57.2 | 65.3 |
| GPT-4o [ 18 ] | 64 | 100 | 69.1 | 65.1 | 65.5 | 50.0 | 68.3 | 63.7 | 63.6 | 49.8 | 71.0 | 55.4 | 58.7 | 27.6 | 73.2 | 59.4 | 53.4 | 58.6 |
| Open-Source Video MLLMs | ||||||||||||||||||
Table 2: Results on OVO-Bench [ 27 ] comprising three categories: i) Real-Time Visual Perception (OCR: Optical Character Recognition, ACR: Action Recognition, ATR: Attribute Recognition, STU: Spatial Understanding, FPD: Future Prediction, OJR: Object Recognition), ii) Backward Tracing (EPM: Episodic Memory, ASI: Action Sequence Identification, HLD: Hallucination Detection), and iii) Forward Active Responding (REC: Repetition Event Count, SSR: Sequential Steps Recognition, CRR: Clues Reveal Responding).
Figure 4: Performance across varying RGB frame ratios. We vary the target rate and evaluate on StreamingBench.
| Model | Gray Tokens/Frame | RGB(%) | All | #VisualToken | |
|---|---|---|---|---|---|
| Full RGB | - | 100 | 77.20 | 100.0% | 100.0% |
| Gray | 256 | 0 | 68.76 | 100.0% | 89.1% |
| Gray | 64 | 0 | 62.08 | 25.0% | 80.4% |
| Gray + TrigRGB | 256 | 8.1 | 73.68 | 100.0% | 95.4% |
| Gray + TrigRGB | 64 | 8.1 | 67.08 | 25.0% | 86.9% |
| Gray + TrigRGB + DT | 64 | 8.1 | 70.72 | 31.1% | 91.6% |
Table 5: Token Efficiency Analysis. denotes the percentage relative to baseline. DT represents Dynamic Token Router, and TrigRGB represents triggered RGB frame selection.
Figure 5: Qualitative Example of the proposed method.
| Method | #Frames | RGB (%) | Video-MME | |||
|---|---|---|---|---|---|---|
| short | medium | long | overall | |||
| VideoChat2-7B [ 21 ] | 16 | 100 | 48.3 | 37.0 | 33.2 | 39.5 |
| LongVA-7B [ 52 ] | 128 | 100 | 61.1 | 50.4 | 46.2 | 52.6 |
| Kangaroo-7B [ 24 ] | 64 | 100 | 66.1 | 55.3 | 46.6 | 56.0 |
| Video-CCAM-14B [ 11 ] | 96 | 100 | 62.2 | 50.6 | 46.7 | 53.2 |
| VideoXL-7B [ 39 ] | 128 | 100 | 64.0 | 53.2 | 49.2 | 55.5 |
Table 6: Performance comparison on Video-MME [ 12 ] .
| Model | #Frames | RGB (%) | OP | CR | CS | ATP | EU | TR | PR | SU | ACP | CT | All |
| Human | - | - | 89.47 | 92.00 | 93.60 | 91.47 | 95.65 | 92.52 | 88.00 | 88.75 | 89.74 | 91.30 | 91.46 |
| Proprietary MLLMs | |||||||||||||
| Gemini 1.5 pro [ 42 ] | 1 fps | 100 | 79.02 | 80.47 | 83.54 | 79.67 | 80.00 | 84.74 | 77.78 | 64.23 | 71.95 | 48.70 | 75.69 |
| GPT-4o [ 18 ] | 64 | 100 | 77.11 | 80.47 | 83.91 | 76.47 | 70.19 | 83.80 | 66.67 | 62.19 | 69.12 | 49.22 | 73.28 |
| Claude 3.5 Sonnet [ 1 ] | 20 | 100 | 80.49 | 77.34 | 82.02 | 81.73 | 72.33 | 75.39 | 61.11 | 61.79 | 69.32 | 43.09 | 72.44 |
| Open-Source Video MLLMs | |||||||||||||
Table 7: Performance comparison on StreamingBench [ 23 ] focusing on Real-Time Visual Understanding tasks. Real-Time Visual Understanding encompasses Object Perception (OP), Causal Reasoning (CR), Clips Summarization (CS), Attribute Perception (ATP), Event Understanding (EU), Text-Rich Understanding (TR), Prospective Reasoning (PR), Spatial Understanding (SU), Action Perception (ACP), and Counting (CT). “RGB(%)” represents the percentage of color frames remaining after triggering, where indicates full color, means all grayscale, and signifies a 91.9% reduction in the number of color frames.
| Model | #Frames | RGB (%) | Real-Time Visual Perception | Backward Tracing | Forward Active Responding | Overall | ||||||||||||
| OCR | ACR | ATR | STU | FPD | OJR | Avg. | EPM | ASI | HLD | Avg. | REC | SSR | CRR | Avg. | Avg. | |||
| Human Agents | - | - | 94.0 | 92.6 | 94.8 | 92.7 | 91.1 | 94.0 | 93.2 | 92.6 | 93.0 | 91.4 | 92.3 | 95.5 | 89.7 | 93.6 | 92.9 | 92.8 |
| Proprietary MLLMs | ||||||||||||||||||
| Gemini 1.5 Pro [ 42 ] | 1 fps | 100 | 87.3 | 67.0 | 80.2 | 54.5 | 68.3 | 67.4 | 70.8 | 68.6 | 75.7 | 52.7 | 62.3 | 35.5 | 74.2 | 61.7 | 57.2 | 65.3 |
| GPT-4o [ 18 ] | 64 | 100 | 69.1 | 65.1 | 65.5 | 50.0 | 68.3 | 63.7 | 63.6 | 49.8 | 71.0 | 55.4 | 58.7 | 27.6 | 73.2 | 59.4 | 53.4 | 58.6 |
| Open-Source Video MLLMs | ||||||||||||||||||
Table 8: Results on OVO-Bench [ 27 ] comprising three categories: i) Real-Time Visual Perception (OCR: Optical Character Recognition, ACR: Action Recognition, ATR: Attribute Recognition, STU: Spatial Understanding, FPD: Future Prediction, OJR: Object Recognition), ii) Backward Tracing (EPM: Episodic Memory, ASI: Action Sequence Identification, HLD: Hallucination Detection), and iii) Forward Active Responding (REC: Repetition Event Count, SSR: Sequential Steps Recognition, CRR: Clues Reveal Responding).
Figure 6: More qualitative examples of ColorTrigger. Our causal trigger selectively injects RGB frames (highlighted in red boxes) when chromatic information is critical for answering queries. (a-b) Success cases: questions requiring color discrimination (bed cover color, car identification) benefit from triggered RGB frames, correcting errors made without ColorTrigger. (c) Failure case: hard insertion of RGB frames disrupts temporal continuity, causing the MLLM to miscount movie clip insertions due to perceived semantic discontinuities (discussed in Sec. 9 ).
More in this topic
Streaming Video Understanding
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.
See all 132 papers