cs.CVNov 13, 2026

V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations

Authors: Keuntae Kim, Dongjun Kim, Yong Suk Choi

Organizations: Department of Computer Science, Hanyang University, Seoul, Republic of Korea

Abstract

World-model video encoders such as V-JEPA learn how scenes change (direction, speed, order, state) from video alone. To answer questions, they are joined to a language model trained on captions and question-answer pairs. But each caption or question names only a few attributes of a clip, so the model learns to retrieve those attributes rather than to read the clip, and the dynamics the encoder carries go unused: on the same items, a linear probe reads speed from frozen V-JEPA features at 87% while the VLM answers at chance. We show where dynamics lives in a frozen encoder: in a subspace shared across videos for predictive encoders such as V-JEPA, but not as a shared component of frame-wise, text-aligned encoders such as SigLIP. We then build counterfactual supervision that makes the language model read it: pixel transformations turn public videos into answer-balanced variants whose answers are set by the transformation and differ only in the change. Trained in a single stage on 4.53M public samples with 32 visual tokens per frame, where open video LLMs use several stages, 9M to 172M samples and 64 to 256 tokens per frame on average, the resulting V-JEPA-LM models (1B to 8B, Qwen3 and Llama) lead every size class in average score across eleven video benchmarks; our 4B model scores 72.1, against 61.1 for the best open 4B and 64.8 for the best open 8B model. A controlled encoder x supervision study shows that the encoder decides what can be read and the supervision what is read: at a matched budget, counterfactual supervision adds 4.5 points on average and 38 on the speed questions of natural benchmarks. Code and models: https://github.com/kkt94/Llama-V-JEPA-LM.

Figures & tables

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

    Aug 27, 2026Lukas Kuhn, Lucas Maes, Giuseppe Serra +4Causal AttentionVideo Representation Learning

  2. Learning Spatiotemporal Sensitivity in Video LLMs via Counterfactual Reinforcement Learning

    May 21, 2026Dazhao Du, Jian Liu, Jialong Qin +7Counterfactual EvaluationVideo-Language Models

  3. Have I Seen Enough? Frozen Video-Language Models Encode Evidence Readiness

    Oct 6, 2026Dan Ben-Ami, Kobi Cohen, Chaim BaskinStreaming Video UnderstandingVideo-Language Model Evaluation