Unifying Video Tasks via Spatiotemporal Analogy

Chia-Hsiang Kao; Belinda Zeng; Bharath Hariharan; Menglin Jia

cs.CVSep 27, 2026

Unifying Video Tasks via Spatiotemporal Analogy

Authors: Chia-Hsiang Kao, Belinda Zeng, Bharath Hariharan, Menglin Jia

Organizations: Cornell University · Meta

Abstract

Adapting video models to new tasks typically requires dedicated data curation and fine-tuning. While visual analogy provides a training-free alternative by specifying tasks in-context, it remains restricted to the image domain. To explore whether analogy-based methods can unify diverse video tasks and generalize to out-of-distribution scenarios, we introduce ViGeo, a framework that extends visual in-context learning to the video domain via spatiotemporal canvas completion. Evaluated on a diverse task taxonomy with a strict train-test split, ViGeo generalizes to unseen video manipulations and zero-shot modalities (e.g., event cameras). Finally, we identify task internalization, where a query format associated with a pretrained task overrides the demonstration, and show that this shortcut can be removed with a small amount of task-unrelated data, highlighting the need to decorrelate prompt format from task identity.

Figures & tables

Figure 1 : ViGeo unifies and generalizes diverse video tasks via video analogy. ViGeo achieves video analogy by inferring a transformation from a demonstration pair and applying it to a novel query. We show that this formulation unifies a highly diverse taxonomy of tasks, including camera motion and view transfer. Additionally, it exhibits strong generalization to unseen guided generation and video manipulation tasks. ViGeo predictions are highlighted in green. Video results are provided in the supplementary material .

Category	Task Family	${\color[rgb]{1,0.3906,0.3047}A}/{\color[rgb]{0.375,0.8516,0.2148}C}$ (input)	${\color[rgb]{0.9961,0.6836,0}B}/{\color[rgb]{0.1758,0.5664,1}D}$ (target)	Layout	#	Split
Camera	Motion transfer	still frame	video @ traj.	$2{\times}2$	1	Train + Eval
Camera	View transfer	video @ pose 1	video @ pose 2	$2{\times}2$	2	Train + Eval
Modal	Modality estimation	rgb video	modality video	$2{\times}2$	5	Train
	Multi-modal quad	rgb video	4-modal grid	$2{\times}2$	1	Train
	Segmentation	rgb video	mask / matted	$2{\times}\{2,3\}$	2	Train
	Guided generation	frozen rgb vid. $\mid$ modal	rgb video	$2{\times}3$	4	Train

Table 1 : Overview of task taxonomy and train/eval protocol. We train on a diverse set of task families that all share a single video analogy canvas

\begin{bmatrix}{\color[rgb]{1,0.3906,0.3047}A}&{\color[rgb]{0.9961,0.6836,0}B}\\ {\color[rgb]{0.375,0.8516,0.2148}C}&{\color[rgb]{0.1758,0.5664,1}D}\end{bmatrix}

: a demonstration pair

({\color[rgb]{1,0.3906,0.3047}A},{\color[rgb]{0.9961,0.6836,0}B})

shows the transformation, and the model must generate

{\color[rgb]{0.1758,0.5664,1}D}

given the query input

{\color[rgb]{0.375,0.8516,0.2148}C}

. Evaluation tasks are strictly held out to ensure the model never sees them during training. For

2{\times}3

layouts, the input spans two grid cells (separated by “

\mid

”). See Appendix C for full per-task schemas.

		Optical Flow	Reconstruction Metrics
Task	Method	EPE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Flow- Guided Generation	FloVD (CVPR 2025)	3.771	12.363	0.240	0.539
	Go-with-the-flow (CVPR 2025)	0.356	19.214	0.491	0.181
	ViGeo	0.185	22.365	0.671	0.106
Point- Guided Generation	Tora (CVPR 2025)	0.612	0 6.923	0.366	0.230
	Wan-Move (NeurIPS 2025)	0.289	17.048	0.365	0.157

Table 2 : Quantitative comparison of our method against strong task-specific baselines on flow-, point-, and event-guided generation. Notably, all three tasks are unseen to our model. The best and second best results are bolded and underlined, respectively.

			Camera Metrics
Task	Dataset	Method	RotErr $\downarrow$	TransErr $\downarrow$	CamErr $\downarrow$
Camera Motion Transfer	DL3DV	CamCloneMaster (SIGGRAPH Asia 2025)	3.367	13.941	15.408
		XFactor (ICLR 2026)	1.066	0 5.378	0 5.761
		ViGeo	2.173	14.612	15.203

Table 3 : Quantitative comparison of our method on the camera_motion_transfer task. The best and second best results are bolded and underlined, respectively.

			Camera Metrics
Task	Dataset	Method	RotErr $\downarrow$	TransErr $\downarrow$	CamErr $\downarrow$
Camera View Transfer	BridgeData-v2	XFactor (ICLR 2026)	36.131	0.857	1.235
	BridgeData-v2	ViGeo	2.382	0.093	0.117
	DROID	XFactor (ICLR 2026)	64.824	1.200	1.959
	DROID	ViGeo	9.149	0.116	0.261

Table 4 : Quantitative result on the camera_view_transfer task. The best results are bolded.

Figure 2 : Generalization to unseen video processing tasks. We evaluate on 12 held-out video processing tasks and report the accuracy between the model output and the target video. Our model generalizes to 7 of 12 unseen tasks, while deblurring and geometric flipping remain challenging, and the model merely copies the input query video.

Figure 3 : Illustration of task internalization . We test an unseen depth-to-flow translation task under three different demo prompts. In prompt 1 (no RGB image provided,

2\times 2

format) and prompt 2 (RGB hint provided, where the first RGB frame appears once), the model succeeds in outputting optical flow visualization video. In prompt 3, where the RGB hint is the first frame repeated across all frames, the model’s outputs collapse to RGB videos.

Figure 4 : Task internalization, and its mitigation. Correlation between the generated output and the query’s RGB video is computed; a higher value means the model outputs RGB instead of the demanded modality. Left: the pretrained model spikes to

0.85

and

0.81

under the repeated RGB hint, in both held-out tasks, and sits near zero in every other condition. Right: after finetuning on a normal-to-edge task that contain neither depth nor optical flow, the spike disappears.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5 : Qualitative results across diverse video tasks. We visualize the model’s predictions for depth-flow dual estimation, flow-guided generation, and camera view transfer. For the demonstration videos in the context, we display only the initial frame ( $t=0$ ) for brevity, while the query and generated target quadrants are shown as temporal rollouts. The results highlight ViGeo ’s ability to synthesize high-fidelity, motion-consistent video while adhering to the geometric and semantic constraints provided by the in-context examples.

			Full Frame			Covisible
Task	Dataset	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
Camera View Transfer	BRIDGE	XFactor (ICLR 2026)	13.268	0.502	0.604	14.279	0.524
	BRIDGE	ViGeo	16.773	0.622	0.299	16.908	0.608
	DROID	XFactor (ICLR 2026)	12.047	0.480	0.666	12.186	0.498
	DROID	ViGeo	14.470	0.564	0.377	14.416	0.569

Appendix

Table 5 : Quantitative comparison of our method on camera view transfer. The best results are bolded.

	DROID ( $N=99$ )		BRIDGE ( $N=100$ )
Method	RotErr $\downarrow$	TransDir $\downarrow$	RotErr $\downarrow$	TransDir $\downarrow$
XFactor (ICLR 2026)	68.92	63.64	39.48	49.12
ViGeo	31.58	18.77	27.32	22.79

Appendix

Table 6 : Quantitative comparison of camera_view_transfer where the demo and query videos are from different hold-out environments.

Figure 6 : Query formats that resemble training inputs trigger task internalization.

Panel of Figure 7 — Figure 7 : Qualitative failure mode analysis of XFactor. We construct a 2x2 grid of cross-domain demo and query pairs. While XFactor easily extracts and applies standard, human-captured camera motions (a, b), it completely collapses when forced to process the extreme roll, pitch, and yaw typical of robotic side-camera setups (c, d).

Figure 8 : Robust perception across modality shifts: in-distribution vs. held-out guidance. Per-checkpoint MAE for modal-guided RGB generation under varying prompt styles. ViGeo successfully consumes out-of-distribution renderings as structural guidance. Left: Edge-guided generation works comparably well with seen (Black BG) and unseen (White BG) edge maps. Middle: Depth-guided generation maintains low error when prompted with unseen Inferno and Turbo colormaps, closely tracking the seen grayscale baseline. Right: Optical flow-guided generation adapts seamlessly to unseen black backgrounds. For reference, the ‘Static baseline’ represents the MAE obtained by trivially repeating the first frame of the ground-truth video.

	Depth-to-Flow		Flow-to-Depth
Method	MAE $\downarrow$	SSIM $\uparrow$	MAE $\downarrow$	SSIM $\uparrow$
ViGeo (base)	0.375	0.335	0.333	0.257
ViGeo (+ finetuning)	0.142	0.818	0.108	0.793

Appendix

Table 7 : Quantitative effect of task internalization under the repeated first-frame hint (

F_{0}

repeated), and its mitigation. Both directions are held out from training; the finetuning data contains neither depth nor optical flow. Best results are bolded.

n{=}55

sequences per cell.

Task	Row Schema	Notes
Training tasks
est_depth	[ $\mid$ depth ]
est_normal	[ $\mid$ normal ]
est_semantic	[ $\mid$ semantic ]	per-pixel semantic class (NYU-40 palette).
est_flow	[ $\mid$ flow ]
est_point	[ $\mid$ points ∗ $\mid$ † ]	∗ static dots on a black image frame, indicating the initial points to be tracked. † RGB video with point tracks overlaid.

Appendix

Table 8 : Per-task schema for Modal-Guided Generation. The demo and query rows share the same cell schema; the rightmost cell is the model’s target (present in the demo, absent and predicted in the query). Convention: (red film icon) = RGB video ;

\cdot

(gray film icon, prefixed with the modality name) = modality video . Analogously, = RGB image (single frame) and

\cdot

= modality image .

Task	Row schema	Notes
Training tasks
zoom-in_static	[full $\mid$ crop]	Static zoom-in to the same region
zoom-in_animated	[full $\mid$ dolly-in]	Animated zoom-in to the same region
zoom-out_static	[crop $\mid$ full]	—
zoom-out_animated	[crop $\mid$ dolly-out]	—
change_params	[orig $\mid$ degraded]	Samples a tuning curve over time and applies it to the original image; variants include saturation, temperature, tone, and chromatic magnitude

Appendix

Table 9 : Per-task schema for Video Manipulation. Demo and query rows share the same cell schema; the rightmost cell is the model’s target. Demo and query use independently sampled degradation parameters, so the model must recover the operation from the demo row rather than memorize a fixed setting. Held-out tasks test new operation families or unseen degradations.

Explore similar work

How Far Are Video Models from True Multimodal Reasoning?
Apr 21, 2026Xiaotian Zhang, Jianhui Wei, Yuan Wang +9Multimodal Reasoning Video Understanding
Zhejiang University · ByteDance
Find, Fix, Reason: Context Repair for Video Reasoning
Apr 17, 2026Haojian Huang, Chuanyu Qin, Yinchuan Li +1Reasoning Skills Large Multimodal Models
The Hong Kong University of Science and Technology (Guangzhou) · Knowin AI · Institute of Information Engineering, Chinese Academy of Sciences
EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning
Nov 23, 2025Yogesh Kulkarni, Pooyan FazliEgocentric Video Fine-Grained Video Understanding
Arizona State University

Apr 21, 2026, cs.CVJ/K move · Enter open · S save

How Far Are Video Models from True Multimodal Reasoning?

Xiaotian Zhang, Jianhui Wei, Yuan Wang, Jie Tan+8

Zhejiang University · ByteDance

Multimodal Reasoning Video Understanding

Open paper AskPDF arXiv

Despite remarkable progress toward general-purpose video models, a critical question remains unanswered: how far are these models from achieving true multimodal reasoning? Existing benchmarks fail to address this question rigorously, as they remain constrained by straightforward task designs and fragmented evaluation metrics that neglect complex multimodal reasoning. To bridge this gap, we introduce CLVG-Bench, an evaluation framework designed to probe video models' zero-shot reasoning capabilities via Context Learning in Video Generation. CLVG-Bench comprises more than 1,000 high-quality, manually annotated metadata across 6 categories and 47 subcategories, covering complex scenarios including physical simulation, logical reasoning, and interactive contexts. To enable rigorous and scalable assessment, we further propose an Adaptive Video Evaluator (AVE) that aligns with human expert perception using minimal annotations, delivering interpretable textual feedback across diverse video context tasks. Extensive experiments reveal a striking answer to our central question: while state-of-the-art (SOTA) video models, such as Seedance 2.0, demonstrate competence on certain understanding and reasoning subtasks, they fall substantially short with logically grounded and interactive generation tasks (achieving success rates <25% and ~0%, respectively), exposing multimodal reasoning and physical grounding as critical bottlenecks. By systematically quantifying these limitations, the proposed method provides actionable feedbacks and a clear roadmap toward truly robust, general-purpose video models. CLVG-Bench and code are released here.

Abstract

Explore similar work

How Far Are Video Models from True Multimodal Reasoning?

Find, Fix, Reason: Context Repair for Video Reasoning

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning