CAST: Modeling Visual State Transitions for Consistent Video Retrieval
Organizations: Google · University of California, Santa Cruz · MIT
Abstract
As video content creation shifts toward long-form narratives, composing short clips into coherent storylines becomes increasingly important. However, prevailing retrieval formulations remain context-agnostic at inference time, prioritizing local semantic alignment while neglecting state and identity consistency. To address this structural limitation, we formalize the task of Consistent Video Retrieval (CVR) and introduce a diagnostic benchmark spanning YouCook2, COIN, and CrossTask. We propose CAST (Context-Aware State Transition), a lightweight, plug-and-play adapter compatible with diverse frozen vision-language embedding spaces. By predicting a state-conditioned residual update () from visual history, CAST introduces an explicit inductive bias for latent state evolution. Extensive experiments show that CAST improves performance on YouCook2 and CrossTask, remains competitive on COIN, and consistently outperforms zero-shot baselines across diverse foundation backbones. Furthermore, CAST provides a useful reranking signal for black-box video generation candidates (e.g., from Veo), promoting more temporally coherent continuations.
Figures & tables
| Dataset | Split Protocol | #Videos | #Step-Clips | #Queries |
| YouCook2 Zhou et al. (2018) | official train; CVR eval on val | 414 | 3,179 | 2,765 |
| COIN Tang et al. (2019) | official train; CVR eval on test | 2,134 | 6,241 | 4,107 |
| CrossTask Zhukov et al. (2019) | video-disjoint 80/20 split | 509 | 2,731 | 2,222 |
| Method | Context | YouCook2 | COIN | CrossTask | Diagnostic | ||||
| Modeling | Acc. | MnR | Acc. | MnR | Acc. | MnR | State | Ident. | |
| CLIP Baseline | Context-Free | 25.03 | 3.60 | 14.10 | 3.91 | 16.83 | 4.15 | 45.52 | 28.90 |
| Late Fusion (Heuristic) | Fixed Weighting | 31.10 | 2.56 | 17.85 | 3.28 | 22.05 | 2.86 | 28.69 | 68.29 |
| Late Fusion (Learned) | Learned Weighting | 36.60 | 2.53 | 44.66 | 2.11 | 25.52 | 2.86 | 40.06 | 76.06 |
| Early Fusion | Feature Concat. | 35.99 | 2.28 | 15.12 | 2.60 | 35.29 | 2.36 | 31.14 | 83.59 |
| CAST (Ours) | State Transition | 44.77 | 2.15 | 40.47 | 2.16 | 47.39 | 2.14 | 53.81 | 74.67 |
| Backbone | Setting | YouCook2 | COIN | CrossTask | Diagnostic | ||||
| Acc. | MnR | Acc. | MnR | Acc. | MnR | State | Ident. | ||
| Category I: Video Foundation Models | |||||||||
| InternVideo2-1B Wang et al. (2024) | Zero-Shot | 36.75 | 2.59 | 17.99 | 3.36 | 20.61 | 3.31 | 65.70 | 30.85 |
| + CAST | 71.68 | 1.48 | 51.03 | 1.90 | 64.36 | 1.71 | 75.43 | 77.77 | |
| VideoPrism-B Zhao et al. (2024) | Zero-Shot | 47.45 | 2.13 | 17.60 | 3.32 | 20.25 | 3.24 | 68.38 | 33.68 |
| + CAST | 75.59 | 1.38 | 51.64 | 1.90 | 62.11 | 1.74 | 76.92 | 77.66 | |
| Fusion | Target | Acc. | State | Ident. |
| Early (Concat) | Direct | 35.99 | 38.92 | 83.58 |
| Early (Concat) | Residual | 38.95 | 43.51 | 81.99 |
| CAST (Ours) | Residual | 44.77 | 51.03 | 78.48 |
| Inference Strategy | Acc. | State | Ident. |
| A. Text Matching ( ) | 25.03 | 50.45 | 25.32 |
| B. Vis. Continuity ( ) | 25.90 | 27.70 | 81.95 |
| C. CAST Prediction ( ) | 42.60 | 50.81 | 75.99 |
| Semantic Ens. (A+C) | 45.46 | 56.71 | 70.38 |
| Full Ens. (A+B+C) | 44.77 | 51.03 | 78.48 |
| Selection Method | Overall Preference | Physical Plausibility | Temporal Logic |
| Standard Text Match | 38.6% | 39.9% | 38.6% |
| CAST Reranking | 55.1% | 50.6% | 52.5% |
| Human Tie | 6.3% | 9.5% | 8.9% |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Backbone | Impl. | #Frames/clip (YC2 / COIN / CT) | Res. | Post-proc. |
| InternVideo2-1B | PyTorch | 8 / 8 / 8 | 224 | mean pool + L2 norm |
| VideoPrism-B | JAX/Flax | 8 / 4 / 4 | 288 | mean pool + L2 norm |
| GME-Qwen2-VL-2B | PyTorch | 3 / 3 / 3 | native | mean pool + L2 norm |
| Qwen3-VL-Embedding-2B | PyTorch | 3 / 3 / 3 | native | mean pool + L2 norm |
| Dataset | Weight training | Coefficient tuning | Final benchmark eval |
| YouCook2 | official train | held-out subset of train | official validation |
| COIN | official train | held-out subset of train | official test |
| CrossTask | crosstask_train.json | held-out subset of train | crosstask_test.json |
| Dataset | State negatives | Identity negatives | Easy negatives |
| YouCook2 | same video, diff step | SBERT top-3 cross-video clips | random diff video |
| COIN | same video, diff step | same recipe + same step, diff video | diff recipe |
| CrossTask | same video, diff step | same task + same step, diff video | diff task |
| Item | Specification |
| Embedding dimension | backbone-dependent (512 for CLIP-B/32) |
| Input normalization | L2 normalization |
| Conditioned path | Linear + LN + ReLU + Dropout(0.1) + Linear |
| Context projections | Linear for query and context |
| Cross-attention | 1 layer, 8 heads |
| Context MLP | LN + Linear + ReLU + Linear |
| Item | Setting |
| Context length | 5 (left-pad with zeros; masked attention) |
| Training epochs | CLIP: 30 / 20 / 50 on YouCook2 / COIN / CrossTask; InternVideo2 / VideoPrism / GME-Qwen2-VL-2B / Qwen3-VL-Embedding-2B: 30 / 30 / 50 |
| Optimizer | AdamW (lr , wd ) |
| Batch size | 512 |
| Temperature | 0.07 |
| Hard negatives (training) | up to 3 state + 3 identity per instance |
Video Retrieval
Video retrieval matches natural-language, visual, or multimodal queries to relevant videos, clips, and moments.
See all 61 papers