cs.CVMar 9, 2026
SaveCAST: Modeling Visual State Transitions for Consistent Video Retrieval
Organizations: Google · University of California, Santa Cruz · MIT
Abstract
As video content creation shifts toward long-form narratives, composing short clips into coherent storylines becomes increasingly important. However, prevailing retrieval formulations remain context-agnostic at inference time, prioritizing local semantic alignment while neglecting state and identity consistency. To address this structural limitation, we formalize the task of Consistent Video Retrieval (CVR) and introduce a diagnostic benchmark spanning YouCook2, COIN, and CrossTask. We propose CAST (Context-Aware State Transition), a lightweight, plug-and-play adapter compatible with diverse frozen vision-language embedding spaces. By predicting a state-conditioned residual update () from visual history, CAST introduces an explicit inductive bias for latent state evolution. Extensive experiments show that CAST improves performance on YouCook2 and CrossTask, remains competitive on COIN, and consistently outperforms zero-shot baselines across diverse foundation backbones. Furthermore, CAST provides a useful reranking signal for black-box video generation candidates (e.g., from Veo), promoting more temporally coherent continuations.
Figures & tables
Figure 1: Given a context clip and instruction, standard retrieval often returns semantically relevant but temporally incoherent clips, yielding State Errors or Identity Errors . In contrast, CAST models the state transition ( ) to retrieve a causally plausible continuation and rerank generation candidates toward more coherent continuations.
Figure 2: Illustration of our CVR benchmark protocol. In contrast to standard global retrieval, our benchmark introduces State Negatives (temporally misaligned clips from the same video) and Identity Negatives (appearance-misaligned clips from different videos) to diagnose consistency failures beyond semantic matching.
Figure 3: Overview of the CAST adapter. CAST operates as a lightweight adapter that aggregates visual history , anchor state , and instruction . Through a dual-path transition predictor, it estimates a residual update that encourages causally consistent state evolution while retaining identity cues through the residual connection.
| Dataset | Split Protocol | #Videos | #Step-Clips | #Queries |
| YouCook2 Zhou et al. (2018) | official train; CVR eval on val | 414 | 3,179 | 2,765 |
| COIN Tang et al. (2019) | official train; CVR eval on test | 2,134 | 6,241 | 4,107 |
| CrossTask Zhukov et al. (2019) | video-disjoint 80/20 split | 509 | 2,731 | 2,222 |
Table 1: CVR benchmark statistics. We report the size of the final CVR evaluation set for each dataset. All settings follow the fixed 1-vs-9 ranking protocol described in Sec. 2.2 . Dataset-specific negative mining rules are summarized in Appendix Table 9 .
| Method | Context | YouCook2 | COIN | CrossTask | Diagnostic | ||||
| Modeling | Acc. | MnR | Acc. | MnR | Acc. | MnR | State | Ident. | |
| CLIP Baseline | Context-Free | 25.03 | 3.60 | 14.10 | 3.91 | 16.83 | 4.15 | 45.52 | 28.90 |
| Late Fusion (Heuristic) | Fixed Weighting | 31.10 | 2.56 | 17.85 | 3.28 | 22.05 | 2.86 | 28.69 | 68.29 |
| Late Fusion (Learned) | Learned Weighting | 36.60 | 2.53 | 44.66 | 2.11 | 25.52 | 2.86 | 40.06 | 76.06 |
| Early Fusion | Feature Concat. | 35.99 | 2.28 | 15.12 | 2.60 | 35.29 | 2.36 | 31.14 | 83.59 |
| CAST (Ours) | State Transition | 44.77 | 2.15 | 40.47 | 2.16 | 47.39 | 2.14 | 53.81 | 74.67 |
Table 2: CVR Benchmark Results (CLIP-B/32). CAST provides the most favorable overall trade-off across datasets and diagnostic metrics, with especially clear gains on state-sensitive retrieval. Diagnostic scores (State Acc. and Ident. Acc.) are averaged across the three benchmark datasets.
| Backbone | Setting | YouCook2 | COIN | CrossTask | Diagnostic | ||||
| Acc. | MnR | Acc. | MnR | Acc. | MnR | State | Ident. | ||
| Category I: Video Foundation Models | |||||||||
| InternVideo2-1B Wang et al. (2024) | Zero-Shot | 36.75 | 2.59 | 17.99 | 3.36 | 20.61 | 3.31 | 65.70 | 30.85 |
| + CAST | 71.68 | 1.48 | 51.03 | 1.90 | 64.36 | 1.71 | 75.43 | 77.77 | |
| VideoPrism-B Zhao et al. (2024) | Zero-Shot | 47.45 | 2.13 | 17.60 | 3.32 | 20.25 | 3.24 | 68.38 | 33.68 |
| + CAST | 75.59 | 1.38 | 51.64 | 1.90 | 62.11 | 1.74 | 76.92 | 77.66 | |
Table 3: Universality Across Backbones. CAST consistently improves over the corresponding zero-shot baseline across diverse frozen video and multimodal embedding models, with both methods operating in the same frozen native vision-language embedding space for each backbone. Diagnostic scores are averaged across the three benchmark datasets.
Figure 4: Qualitative retrieval examples on the CVR benchmark. Given the same procedural context and instruction, context-agnostic retrieval often returns semantically relevant but temporally inconsistent clips, producing either a State Error or an Identity Error . By contrast, CAST retrieves the correct continuation in both cases by modeling state transitions conditioned on visual history.
| Fusion | Target | Acc. | State | Ident. |
| Early (Concat) | Direct | 35.99 | 38.92 | 83.58 |
| Early (Concat) | Residual | 38.95 | 43.51 | 81.99 |
| CAST (Ours) | Residual | 44.77 | 51.03 | 78.48 |
Table 4: Ablation of target formulation and architecture on the YouCook2 validation benchmark. Residual modeling consistently outperforms direct target prediction, and the dual-path CAST design further improves over simple early fusion.
| Inference Strategy | Acc. | State | Ident. |
| A. Text Matching ( ) | 25.03 | 50.45 | 25.32 |
| B. Vis. Continuity ( ) | 25.90 | 27.70 | 81.95 |
| C. CAST Prediction ( ) | 42.60 | 50.81 | 75.99 |
| Semantic Ens. (A+C) | 45.46 | 56.71 | 70.38 |
| Full Ens. (A+B+C) | 44.77 | 51.03 | 78.48 |
Table 5: Inference signal decomposition on the YouCook2 validation benchmark. We analyze the three inference components defined in Sec. 2.4 and their combinations. While the semantic ensemble achieves the highest exact-match accuracy, the full ensemble provides a more balanced trade-off between state discrimination and identity preservation.
Figure 5: Retrieval quality breakdown. We categorize top-1 retrieval outcomes as Exact Match (green), Identity Consistent but State-misaligned (blue), or Identity Inconsistent (gray). CAST yields identity-consistent outcomes in 81.0% of queries (green+blue), substantially improving over the text-only baseline. This differs from the Ident. Acc. metric in Table 5 , which requires the ground-truth clip to rank above all identity negatives.
Figure 6: Effect of context history length ( ). Results are shown on YouCook2, COIN, and CrossTask. Performance improves sharply from to , and then largely saturates as increases further.
| Selection Method | Overall Preference | Physical Plausibility | Temporal Logic |
| Standard Text Match | 38.6% | 39.9% | 38.6% |
| CAST Reranking | 55.1% | 50.6% | 52.5% |
| Human Tie | 6.3% | 9.5% | 8.9% |
Table 6: Guiding Video Generation. Blind human evaluation on 300 prompts from the YouCook2 validation benchmark. For each prompt, Veo generates candidate videos, and we compare the top-ranked outputs selected by standard text matching and CAST reranking. Of the 300 prompts, 158 are non-overlap cases where the two methods select different candidates; we report majority-vote preference rates from annotators on this subset.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Backbone | Impl. | #Frames/clip (YC2 / COIN / CT) | Res. | Post-proc. |
| InternVideo2-1B | PyTorch | 8 / 8 / 8 | 224 | mean pool + L2 norm |
| VideoPrism-B | JAX/Flax | 8 / 4 / 4 | 288 | mean pool + L2 norm |
| GME-Qwen2-VL-2B | PyTorch | 3 / 3 / 3 | native | mean pool + L2 norm |
| Qwen3-VL-Embedding-2B | PyTorch | 3 / 3 / 3 | native | mean pool + L2 norm |
Table 7: Backbone feature extraction settings for the universality experiments. All backbones are frozen. For each backbone, zero-shot retrieval and CAST are both evaluated in the backbone’s native text-video embedding space, while the table below summarizes the corresponding video-side feature extraction settings.
| Dataset | Weight training | Coefficient tuning | Final benchmark eval |
| YouCook2 | official train | held-out subset of train | official validation |
| COIN | official train | held-out subset of train | official test |
| CrossTask | crosstask_train.json | held-out subset of train | crosstask_test.json |
Table 8: Dataset split protocol used in CAST.
| Dataset | State negatives | Identity negatives | Easy negatives |
| YouCook2 | same video, diff step | SBERT top-3 cross-video clips | random diff video |
| COIN | same video, diff step | same recipe + same step, diff video | diff recipe |
| CrossTask | same video, diff step | same task + same step, diff video | diff task |
Table 9: Dataset-specific benchmark mining rules.
| Item | Specification |
| Embedding dimension | backbone-dependent (512 for CLIP-B/32) |
| Input normalization | L2 normalization |
| Conditioned path | Linear + LN + ReLU + Dropout(0.1) + Linear |
| Context projections | Linear for query and context |
| Cross-attention | 1 layer, 8 heads |
| Context MLP | LN + Linear + ReLU + Linear |
Table 10: CAST module specification.
| Item | Setting |
| Context length | 5 (left-pad with zeros; masked attention) |
| Training epochs | CLIP: 30 / 20 / 50 on YouCook2 / COIN / CrossTask; InternVideo2 / VideoPrism / GME-Qwen2-VL-2B / Qwen3-VL-Embedding-2B: 30 / 30 / 50 |
| Optimizer | AdamW (lr , wd ) |
| Batch size | 512 |
| Temperature | 0.07 |
| Hard negatives (training) | up to 3 state + 3 identity per instance |
Table 11: CAST training and inference settings. Optimizer, batch size, temperature, and the overall inference formulation are shared across experiments, while selected training and inference details vary by dataset and backbone setting.
Figure 7: Qualitative example of generation reranking. Given a visual context and next-step instruction, text-only ranking is distracted by semantic similarity and consequently over-ranks procedurally mismatched candidates, whereas CAST more faithfully captures the expected next procedural state and selects a more coherent continuation.
Figure 8: Human evaluation protocol for generation reranking. For each prompt, Veo generates candidate videos from a single context frame and a next-step instruction. Standard text matching and CAST reranking each select one top-ranked output from the same candidate pool, and annotators then compare the two selected videos in a blind A/B setting using three annotators, randomized order, and the options A , B , Tie , or Cannot judge .
More in this topic
Video Retrieval
Video retrieval matches natural-language, visual, or multimodal queries to relevant videos, clips, and moments.
See all 61 papers