Foresight: planning future perception in streaming VLMs without retraining

Ashok Prasad Neupane; Dipan Bartaula; Ankit Belbase; Saugat Adhikari; Samip Ghimire; Saroj Poudel; Binod Bhattarai; Danda Pani Paudel

cs.CVOct 2, 2026

Foresight: planning future perception in streaming VLMs without retraining

Authors: Ashok Prasad Neupane, Dipan Bartaula, Ankit Belbase, Saugat Adhikari, Samip Ghimire, Saroj Poudel, Binod Bhattarai, Danda Pani Paudel

Organizations: Independent Researcher · NAAMII, Nepal · University College London, UK · University of Aberdeen, UK ·

Abstract

Existing streaming vision-language models (VLMs) continuously perceive and reason over visual streams, but their computational pathways remain fixed throughout inference. Consequently, they cannot adapt computation to evolving scene dynamics, where different future events demand different levels and forms of perception. We show that streaming VLMs inherently possess the ability to anticipate the immediate future, and leverage this capability to dynamically configure future computation in a training-free manner. Realizing such anticipatory computation, however, is very challenging: future anticipation must be sufficiently reliable to guide computation, planning must run concurrently with streaming inference, and online reconfiguration must incur negligible overhead. To address these challenges, we introduce FORESIGHT, a dual-stream architecture comprising two Siamese LLMs with shared weights, input encoders, and KV cache. The first LLM continuously processes incoming tokens, while the second runs ahead of the stream to anticipate future context, plan future computation, and generate task responses without interrupting streaming inference. Each plan decides when to reason next, what to check then, and how densely to sample, keeping transient evidence separate from persistent control. The resulting computation plan is executed online through an efficient reconfiguration protocol with schemaguided decoding and lightweight diff-based updates, enabling dynamic adaptation with low overhead. With a frozen Qwen3-VL-8B backbone, FORESIGHT achieves 23.0 mean joint F1 on OmniPro Online evaluation beating strongest trained baseline by 9.5%, while improving the backbone by 6.7 on StreamingBench and 15.4 on OVO-Bench, with the largest gain of 18.7 when evidence arrives later in the video stream. Our source code will be made publicly available.

Figures & tables

Figure 2: Foresight overview. One frozen VLM runs as two twins over a shared KV cache $C_{t}$ . The Ingest LLM (perception) appends every frame admitted by the Vision Gate to $C_{t}$ and never stops. When a check is due, the Think LLM (planning) reads a snapshot of $C_{t}$ and writes a plan $\pi_{t}=(s_{t},u_{t})$ (right panel): an evidence state $s_{t}$ and a control update $u_{t}$ . The Live Executor applies $u_{t}$ along the green control lines and emits the answer once the evidence is sufficient.

Figure 3: Foresight on one stream. Task: “Let me know when a player scores a goal.” Time runs left to right, and the red line marks the goal. The Ingest never pauses (Alg. 1 , lines 3–6), and the cache grows slowly at 1 fps and faster at 8 fps. The Think LLM wakes only at scheduled checks (green bars; line 8), and the interval $\Delta$ is set by the previous plan (line 17). Early plans track only the player and ball. As a shot develops, a plan raises sampling to 8 fps and adds the goalkeeper and net; later plans compact the cache (thus KV-cache drops). Just after the goal, the evidence is sufficient and the answer is emitted; the plan then drops the goalkeeper and net, and sampling returns to 1 fps.

		Perception				Comprehension			Reasoning
Model	Params	Event-	Target-	State-	Snap.-	Cond.-	Cum.-	Event-	Dedup.-	Step-	Mean
Model	Params	Alert	Ground	Monitor	Count	Alert	Count	Narr.	Count	Inst.	Mean
LiveStar ( Yang et al., 2025c )	8B	9.7	0.8	0.0	0.0	14.7	0.0	1.6	0.0	6.0	3.6
MMDuet2 ( Wang et al., 2026 )	3B	12.5	5.3	14.9	11.2	21.4	5.3	3.7	12.7	14.7	11.3
MiniCPM-o 4.5 ( Cui et al., 2026 )	9B	50.1	12.6	9.8	8.0	23.8	9.0	3.0	3.8	1.4	13.5
Dispider † ( Qian et al., 2025 )	7B	2.3	0.0	0.0	0.0	8.5	0.3	4.5	0.1	9.7	2.8

Table 1: OmniPro Online Mode, joint F1 (%) on the subset that does not need audio. Bold is best, underline second best. The propsoed Foresight performs the best, despite being training-free.

Method	FPS	RTVU $\uparrow$	Omni-Source $\uparrow$	Contextual $\uparrow$	Overall $\uparrow$
VideoLLM-Online ( Chen et al., 2024a )	2	35.99	28.45	26.55	32.48
Flash-VStream ( Zhang et al., 2024a )	1	23.23	26.00	24.12	24.04
Dispider ( Qian et al., 2025 )	1	67.63	35.66	33.61	53.12
TimeChat-Online ‡ ( Yao et al., 2025 )	1	75.36	36.96	35.63	58.11
StreamAgent ( Yang et al., 2025a )	1	74.28	36.26	34.62	57.02
Qwen3-VL-8B-Instruct ∗ ( Bai et al., 2025a )	–	74.10	41.90	39.80	59.30

Table 2: StreamingBench, accuracy (%). Bold is best, underline second best.

Method	Frames	Real-Time $\uparrow$	Backward $\uparrow$	Forward $\uparrow$	Overall $\uparrow$
Human	native	93.20	92.30	92.90	92.80
Flash-VStream ( Zhang et al., 2024a )	1 fps	28.37	27.38	45.09	33.61
Dispider ( Qian et al., 2025 )	1 fps	54.55	36.06	34.72	41.78
TimeChat-Online ( Yao et al., 2025 )	1 fps	58.60	42.00	36.40	45.60
StreamForest ( Zeng et al., 2025 )	1 fps	61.20	52.02	53.49	55.57
QueryStream ( Zhang et al., 2026 )	–	61.40	42.10	39.03	47.51

Table 3: OVO-Bench, accuracy (%). Bold is best, underline second best.

Component	Ablation	Time F1	Joint F1
Full Foresight (as reported)		43.7	23.0
Think LLM	Next Check	43.0	18.9
Think LLM	Next Question	43.3	19.4
Think LLM	Next Check + Next Question	43.2	18.6
Vision Gate	FPS	43.5	23.0
Vision Encoder	Keep Classes	42.7	22.7

Table 4: Controller and input-processing ablations on OmniPro Online (%). Plan-field rows remove the named control while keeping everything else fixed. The input-processing row replaces two-frame clips with individual-frame inputs.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Time F1
Method	EA	TG	SM	SC	CA	CC	EN	DC	SI	Overall
Dispider [ Qian et al., 2025 ]	2.3	7.7	8.5	2.7	8.5	7.3	9.5	8.8	15.0	9.6
StreamAgent [ Yang et al., 2025a ]	6.8	8.8	10.5	9.0	11.3	5.7	16.0	17.2	27.3	15.0
QueryStream [ Zhang et al., 2026 ]	8.2	6.9	24.9	7.5	17.6	22.1	19.9	21.1	30.9	17.7
MiniCPM-o 4.5 [ Cui et al., 2026 ]	50.1	53.9	20.9	24.2	23.8	22.5	8.5	7.7	8.3	24.4
Foresight	44.7	47.9	52.6	48.3	33.8	41.7	29.6	41.3	53.2	43.7

Appendix

Table 5: OmniPro Online, per task (%). The three blocks decompose the same emissions: Time F1 counts responses that land inside the

\pm 3

s window, content accuracy how many of those were right, and Joint F1 requires both. The two alert tasks are content-trivial once timed, so their content is 100 by construction. Time F1

\times

content

=

Joint F1 holds for each task, but each source pools the Overall column differently, so it need not multiply out there. Dispider’s content row is recovered from its other two rows task by task, so its Overall is their macro mean.

Method	FPS	OP	CR	CS	ATP	EU	TR	PR	SU	ACP	CT	Avg.
Human	native	89.47	92.00	93.60	91.47	95.65	92.52	88.00	88.75	89.74	91.30	91.46
Gemini 1.5 Pro [ Gemini Team et al., 2024 ]	1	79.02	80.47	83.54	79.67	80.00	84.74	77.78	64.23	71.95	48.70	75.69
GPT-4o [ OpenAI, 2024 ]	64	77.11	80.47	83.91	76.47	70.19	83.80	66.67	62.19	69.12	49.22	73.28
Claude 3.5 Sonnet [ Anthropic, 2024 ]	20	80.49	77.34	82.02	81.73	72.33	75.39	61.11	61.79	69.32	43.09	72.44
Video-LLaMA2-7B [ Cheng et al., 2024 ]	32	55.86	55.47	57.41	58.17	52.80	43.61	39.81	42.68	45.61	35.23	49.52
VILA-1.5-8B [ Lin et al., 2024a ]	14	53.68	49.22	70.98	56.86	53.42	53.89	54.63	48.78	50.14	17.62	52.32

Appendix

Table 6: StreamingBench Real-Time Visual Understanding, per task (accuracy, %). The first two blocks are reference points rather than streaming systems, taken from Lin et al. [2024b] ; a plain frame count means uniform sampling rather than a streaming rate. TimeChat-Online’s keep-rate variants are from Yao et al. [2025] and the Qwen3-VL-8B and MiniCPM-o 4.5 rows from Lu et al. [2026] .

Method	ER	SCU	SD	MA	Avg.	ACU	MCU	SQA	PO	Avg.	Overall
	Omni-Source					Contextual
Flash-VStream [ Zhang et al., 2024a ]	25.91	24.90	25.60	28.40	26.00	24.80	25.20	26.80	1.96	24.12	24.04
VideoLLM-Online [ Chen et al., 2024a ]	31.20	26.51	24.10	32.00	28.45	24.19	29.20	30.80	3.92	26.55	32.48
Dispider [ Qian et al., 2025 ]	35.46	25.26	38.57	43.34	35.66	39.62	27.65	34.80	25.34	33.61	53.12
StreamAgent [ Yang et al., 2025a ]	35.86	26.26	38.87	44.04	36.26	39.72	30.25	39.60	28.90	34.62	57.02
TimeChat-Online ‡ [ Yao et al., 2025 ]	36.62	26.75	38.55	45.90	36.96	41.45	32.27	40.55	28.25	35.63	58.11

Appendix

Table 7: StreamingBench Omni-Source and Contextual, per task (accuracy, %). These are the systems with results on the full 18-task evaluation; ‡ marks our own run under the official protocol. Baseline Contextual averages include Proactive Output, as in their sources. For Foresight , the Contextual average and overall cover the three multiple-choice tasks, since Proactive Output scores timing rather than answer choice and is reported separately (46.3, 95% CI 39.1–53.7; Appendix D.2 ); averaged over all four tasks, its Contextual score is 44.50.

Method	Frames	OCR	ACR	ATR	STU	FPD	OJR	Avg.
Human	native	94.0	92.6	94.8	92.7	91.1	94.0	93.20
Gemini 1.5 Pro [ Gemini Team et al., 2024 ]	1 fps	85.91	66.97	79.31	58.43	63.37	61.96	69.32
GPT-4o [ OpenAI, 2024 ]	64	69.80	64.22	71.55	51.12	70.30	59.78	64.46
LLaVA-Video-7B [ Zhang et al., 2025c ]	64	69.13	58.72	68.83	49.44	74.26	59.78	63.52
LLaVA-OneVision-7B [ Li et al., 2025a ]	64	66.44	57.80	73.28	53.37	71.29	61.96	64.02
Qwen2-VL-7B [ Wang et al., 2024b ]	64	60.40	50.46	56.03	47.19	66.34	55.43	55.98

Appendix

Table 8: OVO-Bench Real-Time Visual Perception, per task (accuracy, %). The second block lists reference models rather than streaming systems, from the current (v2) release of Niu et al. [2025] ; Qwen3-VL-8B and MiniCPM-o 4.5 are from Lu et al. [2026] .

	Backward Tracing				Forward Active Responding
Method	EPM	ASI	HLD	Avg.	REC	SSR	CRR	Avg.	Overall
Human	92.6	93.0	91.4	92.30	95.5	89.7	93.6	92.90	92.80
Gemini 1.5 Pro [ Gemini Team et al., 2024 ]	58.59	76.35	52.64	62.54	35.53	74.24	61.67	57.15	63.00
GPT-4o [ OpenAI, 2024 ]	57.91	75.68	48.66	60.75	27.58	73.21	59.40	53.40	59.54
LLaVA-Video-7B [ Zhang et al., 2025c ]	56.23	57.43	7.53	40.40	34.10	69.95	60.42	54.82	52.91
LLaVA-OneVision-7B [ Li et al., 2025a ]	54.21	55.41	21.51	43.71	25.64	67.09	58.75	50.50	52.74

Appendix

Table 9: OVO-Bench temporal-reasoning breakdown (accuracy, %). Reference rows follow the same provenance as Table 8 , except VideoLLM-Online’s Forward Active Responding and overall, which the original release does not report and which come from our own run.

Figure 4: Anticipation quality against lead time. Each curve shows how well a frozen VLM anticipates an upcoming event when it is asked

T

seconds before the event, using only the stream seen so far. Scores are bracketed between two references: Blind (0.2168), a baseline that never sees the video, and Oracle (0.508), which observes the event itself. Dashed lines are individual backbones and the solid red line pools them. The pooled score is well above the blind baseline for near-future events and falls steadily as the horizon grows, approaching it by

T=30

s; individual backbones are noisier, and Qwen2.5-Omni-7B reaches the blind baseline from

T=20

s. Anticipation is therefore most reliable over the short horizons at which Foresight schedules its next check.

Figure 5: Total compute against check rate. Each curve is the compute a design spends on a 300 s stream. Designs that re-read the stream at every check grow quickly with

q

: without a shared KV cache every check prefills the whole history again, and with a repeated prompt every check pays for the prompt on top of the plan. Ingesting at a fixed rate without pruning costs the same at every

q

, so it appears as a flat line set by how many frames are admitted. Foresight stays almost flat across its operating region: nearly all of its cost is the one-time ingest of the video, and each check adds only a short probe. The shaded bands and the distributions

p(q)

and

p_{\mathrm{dense}}(q)

show where its check rate falls in typical and event-dense scenes; the star marks the ideal operating point of one check per event.

Figure 6: Compute above the oracle. The same model, with the oracle’s compute subtracted from every design. Because the oracle shares the ingest pipeline, what remains is only the work a design spends beyond the least it could, which makes differences that are invisible in Figure 5 easy to read. Removing adaptive frames or pruning adds a fixed cost from extra visual tokens, so these curves are flat. The controller variants grow with

q

, and their order shows what each saving buys: a full plan at every check costs most, then a diff at every check, then a separate probe followed by a plan, and least of all Foresight , which folds the probe into the plan and writes a diff only when evidence arrives. In its operating region Foresight wastes about half as much as the probe–plan hybrid, a quarter as much as a diff at every check, and about a hundredth as much as the designs without adaptive frames or pruning.

Task	Abbr.	Threshold	Mode	Refractory (s)
Target-Ground	TG	0.50	edge	300
Event-Alert	EA	0.45	edge	600
State-Monitor	SM	0.80	level	7
Conditional-Alert	CA	0.98	level	10
Step-Instruction	SI	0.01	level	7
Snapshot-Count	SC	0.985	edge	600

Appendix

Table 10: Response-gate settings on OmniPro Online. Threshold on

p_{\mathrm{hit},t}

, gate mode, and refractory interval for each task, fitted on the calibration split.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 721 papers

Mar 3, 2026

Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models

Large Vision Language Models (LVLMs) exhibit strong Chain-of-Thought (CoT) capabilities, yet most existing paradigms assume full-video availability before inference, a batch-style process misaligned with real-world video streams where information arrives sequentially. Motivated by the streaming nature of video data, we investigate two streaming reasoning paradigms for LVLMs. The first, an interleaved paradigm, alternates between receiving frames and producing partial reasoning but remains constrained by strictly ordered cache updates. To better match streaming inputs, we propose Think-as-You-See (TaYS), a unified framework enabling true concurrent reasoning. TaYS integrates parallelized CoT generation, stream-constrained training, and stream-parallel inference. It further employs temporally aligned reasoning units, streaming attention masks and positional encodings, and a dual KV-cache that decouples visual encoding from textual reasoning. We evaluate all paradigms on the Qwen2.5-VL family across representative video CoT tasks, including event dynamics analysis, causal reasoning, and thematic understanding. Experiments show that TaYS consistently outperforms both batch and interleaved baselines, improving reasoning performance while substantially reducing time-to-first-token (TTFT) and overall reasoning delay. These results demonstrate the effectiveness of data-aligned streaming reasoning in enabling efficient and responsive video understanding for LVLMs. We release our code at https://github.com/EIT-NLP/StreamingLLM/tree/main/TaYS

Jialiang Zhang, Junlong Tong, Junyan Lin +4

Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ocean University of China · Shanghai Jiao Tong University +3

VLM Reasoning Streaming Inference Streaming Video Understanding Large Vision-Language Models Efficient VLM Inference

Ask PDF

Jun 7, 2026

Harnessing Streaming Video in the Wild

Vision-Language Models (VLMs) are increasingly required to process unbounded video streams in applications such as video-call assistants, live commentary, and embodied robots. An ideal streaming system should support proactive interaction, long-horizon memory, and real-time processing, while resting on a VLM backbone capable of handling diverse in-the-wild streaming tasks. However, existing VLMs excel at offline video understanding but fall short in streaming capabilities and lack dedicated infrastructure for streaming deployment. We address this gap on three fronts. (i) For backbone capability, we construct Streaming-Train-248K, a streaming dataset paired with a novel training objective for adapting VLMs to streaming interaction and understanding. (ii) For real-world deployment, we introduce Streaming Harness, a plug-and-play system that endows any VLM with three core abilities: proactive interaction (per-second response decisions), long-term memory (12-hour context retention), and real-time processing (sub-second latency). (iii) To drive continued community progress on streaming capabilities, we design Streaming-Eval, a benchmark that reflects models' capabilities across diverse in-the-wild scenarios. Extensive experiments demonstrate consistent gains from our approach across all core capabilities required for streaming video understanding. We will open-source our data, code, and benchmark to advance the community's shift from offline video understanding to deployable streaming intelligence.

Dingyu Yao, Shuhuan Gu, Qingyi Si +8

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

VLM Evaluation Streaming Video Understanding VLM Adaptation Streaming Inference

Ask PDF

Apr 7, 2026

CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference

Continuous inference over concurrent video streams imposes substantial compute and memory demands on vision-language model (VLM) serving. Streaming inference uses sliding windows to maintain a bounded context of recent video, but processing each window independently repeats visual encoding and large language model (LLM) prefilling for similar and overlapping content. Existing optimizations provide limited coordination across these stages and often rely on model-specific training, profiling, or model-generated signals. We present CodecSight, a streaming VLM serving system that uses codec metadata as shared runtime guidance across visual encoding and LLM prefilling, without model-specific training or offline profiling. Codec-derived change signals guide patch pruning before visual encoding, reducing both visual computation and the number of downstream visual tokens. Codec-defined frame types guide selective key-value (KV) refresh across windows, while positional correction enables reuse of the remaining cached keys. Across three VLMs and four video workloads, our vLLM-based implementation supports up to

3.3\times

as many concurrent streams and achieves up to a

5.3\times

speedup in average time-to-first-token relative to the state-of-the-art baselines. It also reduces executed FLOPs by up to 93%, with a maximum task-quality decrease of 4.64 percentage points.

Yulin Zou, Wenyan Chen, Yan Chen +6

NTU Singapore Singapore · Beihang University China · A*STAR IAIC Singapore +1

Efficient VLM Inference VLM Inference Streaming Inference

Ask PDF

Abstract

Vision-Language Models

Explore similar work

Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models

Harnessing Streaming Video in the Wild

CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference