V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations
Organizations: Department of Computer Science, Hanyang University, Seoul, Republic of Korea
Abstract
World-model video encoders such as V-JEPA learn how scenes change (direction, speed, order, state) from video alone. To answer questions, they are joined to a language model trained on captions and question-answer pairs. But each caption or question names only a few attributes of a clip, so the model learns to retrieve those attributes rather than to read the clip, and the dynamics the encoder carries go unused: on the same items, a linear probe reads speed from frozen V-JEPA features at 87% while the VLM answers at chance. We show where dynamics lives in a frozen encoder: in a subspace shared across videos for predictive encoders such as V-JEPA, but not as a shared component of frame-wise, text-aligned encoders such as SigLIP. We then build counterfactual supervision that makes the language model read it: pixel transformations turn public videos into answer-balanced variants whose answers are set by the transformation and differ only in the change. Trained in a single stage on 4.53M public samples with 32 visual tokens per frame, where open video LLMs use several stages, 9M to 172M samples and 64 to 256 tokens per frame on average, the resulting V-JEPA-LM models (1B to 8B, Qwen3 and Llama) lead every size class in average score across eleven video benchmarks; our 4B model scores 72.1, against 61.1 for the best open 4B and 64.8 for the best open 8B model. A controlled encoder x supervision study shows that the encoder decides what can be read and the supervision what is read: at a matched budget, counterfactual supervision adds 4.5 points on average and 38 on the speed questions of natural benchmarks. Code and models: https://github.com/kkt94/Llama-V-JEPA-LM.
Figures & tables
| Family | Construction of | Answer rule | Source videos | Rows | Pair% |
| Sign flip | State change played backwards | action inverse (open/close) | Charades-Ego, EPIC, SSv2 | 49k | 3 |
| Object direction, clip mirrored or reversed | direction remapped | SSv2 | 37k | 0 | |
| Camera pan, tilt or zoom on static footage, or reversed | move remapped | CLEVRER, Charades-Ego, SSv2 | 23k | 0 | |
| Two clips joined, each optionally reversed | direction A then B | SSv2 | 7k | 0 | |
| Order | Two events in sequence or stacked at once | first / second / simultaneous | LLaVA-Video, SA-V, SSv2 | 65k | 0 |
| Segment matching a two-action caption | matching segment | HoloAssist | 20k | 80 |
| reversal | speed | mirror | speed readout | |||
|---|---|---|---|---|---|---|
| Encoder | proto 8 | probe | ||||
| SigLIP so400m | 0 | – | .0016 | .0014 | 50.0 | 73.5 |
| ViCLIP (text-aligned) | .052 | .0012 | .042 | .0019 | 60.1 | 70.3 |
| VideoMAE (pixel pred.) | .502 | .0032 | .632 | .045 | 100.0 | 76.8 |
| V-JEPA ViT-L (1B, 2B) | .064 | .037 | .080 | .031 | 66.3 | 81.2 |
| V-JEPA ViT-g (3B–8B) | .108 | .039 | .167 | .074 | 77.2 | 84.7 |
| linear probe on the stage | VLM | |||||||
|---|---|---|---|---|---|---|---|---|
| Encoder | Training | frozen | bridge | L4 | L14 | L18 | L28 | answer |
| SigLIP | caps+QA | 36.4 | 38.7 | 36.8 | 35.9 | 35.7 | 34.7 | 33.1 |
| SigLIP | +CF | 36.3 | 39.1 | 37.5 | 37.1 | 35.9 | 35.8 | 34.4 |
| V-JEPA | caps+QA | 86.7 | 83.5 | 53.8 | 47.2 | 49.4 | 46.9 | 33.7 |
| V-JEPA | +CF | 86.8 | 91.2 | 66.6 | 78.9 | 96.3 | 96.5 | 92.8 |
| Model | MVB | PT | TVB | TC | TOM | MoB | NQA | MVP | TB | SSv2 | VG-T | VG-V | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2B class | |||||||||||||
| Qwen3-VL-2B | 61.0 | 65.1 | 49.8 | 67.4 | 30.3 | 57.5 | 77.1 | 32.3 | 65.9 | 59.2 | 36.2 | 27.6 | 52.45 |
| InternVL3.5-2B | 66.8 | 64.2 | 50.9 | 65.7 | 31.0 | 56.2 | 76.1 | 28.8 | 71.3 | 57.0 | 40.0 | 21.0 | 52.42 |
| PLM-1B | 68.0 | 69.8 | 48.2 | 60.5 | 24.5 | 51.7 | 79.8 | 30.9 | 58.1 | 61.7 | 33.2 | 29.6 | 51.33 |
| Apollo-1.5B | 61.1 | 63.9 | 47.9 | 60.3 | 27.8 | 51.1 | 78.8 | 26.0 | 55.8 | 56.0 | 38.8 | 22.4 | 49.16 |
| VideoLLaMA3-2B | 57.9 | 62.9 | 44.0 | 64.1 | 24.0 | 56.3 | 75.4 | 23.1 | 64.1 | 50.0 | 38.0 | 25.2 | 48.75 |
| Model | Tok. | Samples | MVB | PT | TVB | TC | TOM | MVP | Avg |
|---|---|---|---|---|---|---|---|---|---|
| V-JEPA 2 (L3.1-8B) | 9,216 | 88.5M | 73.5 | 84.0 | 60.6 | 76.9 | 40.3 | 44.5 | 63.3 |
| V-JEPA 2.1 (L3.1-8B) | – | 72.5M | 75.4 | 83.1 | 62.8 | 74.0 | 38.0 | 43.2 | 62.8 |
| Ours-8B-Q (Qwen3-8B) | 1,024 | 4.53M | 72.8 | 77.8 | 75.3 | 79.4 | 56.5 | 58.1 | 70.0 |
| Ours-8B-L (L3.1-8B) | 1,024 | 4.53M | 71.9 | 75.0 | 71.8 | 77.6 | 55.5 | 57.6 | 68.2 |
| 12-score avg | abilities at 8k (natural benchmarks) | held-out sets at 8k | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Encoder | Supervision | 8k | final | speed | direction | camera | knowledge | 6-set avg | unseen ratios | rev. cons. |
| SigLIP so400m | caps+QA | 53.46 | 58.52 | 34.0 | 39.9 | 36.4 | 63.5 | 47.1 | 35.0 | 19.4 |
| SigLIP so400m | +CF | 56.77 | 62.13 | 45.4 | 40.0 | 36.4 | 62.6 | 59.3 | 34.8 | 19.7 |
| V-JEPA ViT-L | caps+QA | 56.65 | 61.53 | 32.4 | 59.7 | 40.5 | 64.1 | 58.6 | 33.5 | 32.3 |
| V-JEPA ViT-L | AoT only | 56.37 | 61.17 | 33.0 | 55.7 | 43.9 | 63.6 | 60.8 | 31.9 | 30.8 |
| V-JEPA ViT-L | +CF | 61.18 | 66.02 | 65.3 | 63.3 | 47.4 | 63.7 | 91.3 | 91.5 | 36.8 |
| new | new corpus | new comb. | new | succ. | ||||
| Model | ratio | spd | dir | dir | spd | cam. | avg | play |
| Qwen3-VL-2B | 37.0 | 45.5 | 35.2 | 56.5 | 44.6 | 43.9 | 43.8 | 33.1 |
| Qwen3-VL-4B | 35.9 | 50.5 | 40.7 | 73.3 | 50.2 | 49.4 | 50.0 | 36.9 |
| Qwen3-VL-8B | 36.7 | 50.8 | 41.9 | 78.4 | 68.0 | 57.1 | 55.5 | 51.9 |
| V-JEPA, no CF | 33.3 | 44.3 | 69.8 | 97.1 | 49.1 | 92.4 | 64.3 | 39.4 |
| Ours-2B-Q | 98.7 | 98.5 | 75.7 | 98.5 | 98.9 | 99.8 | 95.0 | 93.9 |
| held-out speed | content | |||||
|---|---|---|---|---|---|---|
| Removed subspace | energy | ratio | comb. | TOM | NQA | PT |
| none | 0 | 98.7 | 98.9 | 50.9 | 74.0 | 75.3 |
| speed directions (8) | 16.6 | 91.5 | 65.7 | 46.4 | 73.2 | 73.8 |
| random, same energy (24) | 16.2 | 96.7 | 97.2 | 49.9 | 73.2 | 74.3 |
| top content directions (24) | 43.6 | 95.0 | 84.3 | 44.8 | 72.4 | 72.0 |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Work | Training signal | Where the answer comes from | Encoder | What is analysed |
|---|---|---|---|---|
| Pretext tasks [ 45 , 19 , 65 , 68 , 7 , 71 , 61 ] | order, arrow-of-time or speed prediction | the transformation | trained by the task | learned features |
| Tracing the Arrow of Time [ 23 ] | arrow-of-time questions, time-preserving connector | reversal | video- vs. frame-centric | information flow, encoder to answer |
| Which Way Did It Move? [ 32 ] | projector objective on frame-feature deltas (DeltaDirect), direction instructions | 2-D motion direction | – | direction readable at each stage, binding gap, direction concept vectors |
| TIME [ 64 ] | answer-balanced temporal instructions, five dimensions | constructed instructions; reversal for direction | – | benchmark gains |
| SynRL [ 28 ] | CoT fine-tuning and RL on rendered shapes | the rendering program | – | synthetic-to-real transfer |
| Ours | 14 constructions in 6 families on public videos, added to captions and QA | the transformation of the same clip, so content cancels | frozen predictive vs. frame-wise | change structure ( , ), stage-wise probes, subspace removal, encoder supervision |
| Source | Construction | Rows | % mix | paired % | Example question (real row) |
| Sign flip (12 sources, 117,066 rows, 1.47% of the mixture) | |||||
| tq4_odir_ssv2 | object direction; clip mirrored or reversed with the answer remapped | 16,116 | 0.15 | 0 | In which direction does the roll move? |
| tq4_cdir_ssv2 | camera direction under mirroring / reversal | 5,895 | 0.05 | 0 | What is the camera motion in the video? |
| tq4_dir2_ssv2 | two concatenated clips, each optionally reversed: direction A then B | 7,204 | 0.07 | 0 | How would you describe the bowl’s directional movement? |
| tq4_state_ssv2 | state-change action reversed: open/close, put/take | 24,659 | 0.23 | 0 | Which action is performed in the video? |
| chego_rev | egocentric actions with reversible verbs, clip reversed | 16,780 | 0.26 | 7 | Which action does the person perform? |
| Model | Language model | Encoder | Batch/GPU GPUs | Global | Steps | Samples | LR (LLM) | EFLOPs |
|---|---|---|---|---|---|---|---|---|
| Ours-2B-Q | Qwen3-1.7B | ViT-L | 12 14 | 168 | 27,382 | 4.60M | 1.05e-5 | 121 |
| Ours-1B-L | Llama-3.2-1B-Instr. | ViT-L | 6 28 | 168 | 27,382 | 4.60M | 1.05e-5 | 104 |
| Ours-4B-Q | Qwen3-4B-Instr.-2507 | ViT-g | 5 30 | 150 | 30,187 | 4.53M | 1.2e-5 | 314 |
| Ours-3B-L | Llama-3.2-3B-Instr. | ViT-g | 10 30 | 300 | 15,094 | 4.53M | 1.7e-5 | 277 |
| Ours-8B-Q | Qwen3-8B | ViT-g | 5 30 | 150 | 30,187 | 4.53M | 8e-6 | 437 |
| Ours-8B-L | Llama-3.1-8B-Instr. | ViT-g | 5 30 | 150 | 30,187 | 4.53M | 8e-6 | 426 |
| Model | H100s | Global batch | Steps | Samples | Samples/s | Hours | H100-hours | Peak mem. (GB) | Loss, first 1% | Loss, last 5% |
|---|---|---|---|---|---|---|---|---|---|---|
| Ours-2B-Q | 14 | 168 | 27,382 | 4.60M | 78 | 16.4 | 229 | 46 | 1.53 | 0.41 |
| Ours-1B-L | 28 | 168 | 27,382 | 4.60M | 118 | 10.8 | 303 | 18 | 1.31 | 0.43 |
| Ours-4B-Q | 30 | 150 | 30,187 | 4.53M | 76 | 16.6 | 499 | 44 | 1.41 | 0.39 |
| Ours-3B-L | 30 | 300 | 15,094 | 4.53M | 88 | 14.3 | 429 | 51 | 1.51 | 0.39 |
| Ours-8B-Q | 30 | 150 | 30,187 | 4.53M | 58 | 21.6 | 648 | 33 | 1.36 | 0.38 |
| Ours-8B-L | 30 | 150 | 30,187 | 4.53M | 63 | 19.9 | 596 | 32 | 1.37 | 0.40 |
| Samples (M) | 2.7 | 3.0 | 3.3 | 3.6 | 3.9 | 4.2 | 4.5 | 4.53 |
|---|---|---|---|---|---|---|---|---|
| Ours-4B-Q | 69.97 | 70.24 | 69.98 | 71.43 | 71.41 | 71.73 | 72.18 | 72.14 |
| Ours-3B-L | 66.40 | 67.14 | 67.39 | 68.33 | 68.73 | 68.54 | 68.92 | 68.97 |
| Ours-8B-Q | 68.66 | 68.47 | 69.28 | 69.89 | 70.12 | 69.92 | 70.23 | 70.11 |
| Ours-8B-L | 68.40 | 68.82 | 68.79 | 69.78 | 70.41 | 70.59 | 70.61 | 70.51 |
| Model | MVBench | PerceptionTest | TVBench | TempCompass | TOMATO | MotionBench | NExT-QA |
|---|---|---|---|---|---|---|---|
| 2B class | |||||||
| Qwen3-VL-2B | 61.7 (61.0) | – (65.1) | – (49.8) | – (67.4) | – (30.3) | – (57.5) | – (77.1) |
| InternVL3.5-2B | 65.9 (66.8) | – (64.2) | – (50.9) | – (65.7) | – (31.0) | – (56.2) | – (76.1) |
| PLM-1B | 70.1 (68.0) | 72.7 t (69.8) | 50.4 (48.2) | 64.6 (60.5) | 25.5 (24.5) | 52.2 (51.7) | 80.3 (79.8) |
| VideoLLaMA3-2B | 65.5 (57.9) | 68.0 t (62.9) | – (44.0) | 63.4 (64.1) | – (24.0) | – (56.3) | 81.1 (75.4) |
| Apollo-1.5B | – (61.1) | 61.0 (63.9) | – (47.9) | 60.8 (60.3) | – (27.8) | – (51.1) | – (78.8) |
| Benchmark | Items | Shared | Shared source videos |
|---|---|---|---|
| MVBench | 4,000 | 60 (1.5%) | Charades (STAR) clips also in LLaVA-Video-178K |
| TVBench | 2,525 | 57 (2.3%) | the same Charades clips; PerceptionTest train |
| TemporalBench | 2,765 | 30 (1.1%) | YouTube videos also in Kinetics-700 |
| The other nine sets | 26,198 | 0 | – |
| All twelve | 35,488 | 147 (0.41%) |
| Model | MVB | PT | TVB | TC | TOM | MoB | NQA | MVP | TB | SSv2 | VG-T | VG-V | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 E2B | 39.8 | 46.3 | 36.4 | 51.5 | 18.7 | 41.1 | 51.4 | 8.1 | 61.4 | 37.0 | 24.0 | 22.4 | 36.51 |
| Gemma 4 E4B | 46.5 | 52.0 | 41.5 | 57.4 | 22.8 | 49.4 | 64.7 | 16.5 | 61.0 | 43.8 | 26.4 | 24.0 | 42.17 |
| Set | Items | Source | What is new |
|---|---|---|---|
| Speed ratio | 540 | SSv2 validation | relative speed of a clip stacked over itself at span ratios 1.6, 2.5, 4.0 (training: equal, 2:1 and 3:1, and 1.0–1.4 in the fine-motion source; none of the three ratios occurs), plus the equal item of each clip |
| Domain, speed | 600 | Kinetics-700 validation | relative speed (2:1, 3:1, equal) and acceleration on a corpus never used by the speed family |
| Domain, direction | 540 | Kinetics-700 validation | camera direction labeled by optical flow, original / reversed / mirrored, on a corpus unused for this family |
| Combo, direction | 584 | SSv2 validation | mirror and reversal together (the answer is the original direction), with the three trained variants of the same clip |
| Combo, speed | 540 | SSv2 validation | acceleration played backwards (accelerating becomes decelerating), plus freeze |
| Camera, domain | 490 | NVIDIA PhysX simulation | synthetic camera moves on a simulator never used for camera rows |
| Model | ratio | dom. spd | dom. dir | comb. dir | comb. spd | camera | avg |
|---|---|---|---|---|---|---|---|
| Open models | |||||||
| Qwen3-VL-2B | 37.0 | 45.5 | 35.2 | 56.5 | 44.6 | 43.9 | 43.8 |
| Qwen3-VL-4B | 35.9 | 50.5 | 40.7 | 73.3 | 50.2 | 49.4 | 50.0 |
| InternVL3.5-4B | 33.9 | 46.7 | 30.7 | 61.1 | 53.0 | 45.1 | 45.1 |
| Molmo2-4B | 38.3 | 50.5 | 29.1 | 59.4 | 51.1 | 32.7 | 43.5 |
| PLM-3B | 35.9 | 49.8 | 35.7 | 64.9 | 58.5 | 44.9 | 48.3 |
| inside training range | beyond | ||
| Model | 1.6 | 2.5 | 4.0 |
| Controlled 2B arms, 8k | |||
| SigLIP +CF | 38.3 | 35.6 | 30.6 |
| V-JEPA caps+QA | 32.8 | 34.4 | 33.3 |
| V-JEPA temporal QA in place of CF | 37.8 | 36.7 | 33.9 |
| V-JEPA +CF | 88.9 | 91.7 | 93.9 |
| Model | Single | Pair |
|---|---|---|
| Open models | ||
| Qwen3-VL-2B | 33.1 | 8.9 |
| Qwen3-VL-4B | 36.9 | 1.1 |
| InternVL3.5-4B | 24.2 | 0.6 |
| Molmo2-4B | 19.4 | 4.4 |
| PLM-3B | 29.2 | 2.8 |
| vision encoder | connector | LLM prefill | first token | decode | 720p clip (ms) | |||||
| Model | vis. tokens | TFLOPs | ms | ms | TFLOPs | ms | ms | ms/token | LLM prefill | first token |
| 2B class | ||||||||||
| Qwen3-VL-2B | 1,920 / 11,520 | 5.0 | 35 | in enc. | 7.0 | 31 | 66 | 18.2 | 154 | 308 |
| InternVL3.5-2B | 8,192 | 23.2 | 55 | 0.4 | 40.5 | 104 | 159 | 18.4 | 104 | 159 |
| PLM-1B | 8,192 | 22.2 | 87 | 1.8 | 25.2 | 57 | 146 | 9.3 | 57 | 146 |
| Ours-2B-Q | 1,024 | 13.9 | 60 | 0.8 | 3.3 | 22 | 83 | 20.0 | 21 | 82 |
| Encoder | Transf. | proto 8 | proto all | |||
|---|---|---|---|---|---|---|
| SigLIP so400m | reversal | 0 | – | – | 50.0 | 50.0 |
| speed | .016 | .0016 | .18 | 50.0 | 50.3 | |
| mirror | .066 | .0014 | .18 | 49.8 | 49.0 | |
| V-JEPA ViT-L | reversal | .064 | .037 | .12 | 58.8 | 63.7 |
| speed | .055 | .080 | .40 | 66.3 | 71.0 | |
| mirror | .093 | .031 | .031 | 58.7 | 70.4 |
| (frozen) | CF gain at 8k | |||||||
|---|---|---|---|---|---|---|---|---|
| Family | Transformation | Theory | Prediction | V-JEPA | SigLIP | V-JEPA | SigLIP | Ability |
| Speed, rate | rescaling | Proposition 1 (i); measured | large on V-JEPA, small on SigLIP | .080 | .0016 | speed | ||
| Sign flips | reversal, camera moves | Proposition 2 : ; Proposition 1 (i): | large on V-JEPA | .037 | – | camera | ||
| state change | ||||||||
| Sign flips | mirror | Proposition 2 ; Proposition 1 (ii) | large on V-JEPA | .031 | .0014 | direction | ||
| Event order | segment permutation | local receptive field: | equal on both | – | – | order | ||
| Step | MVB | PT | TVB | TC | TOM | MoB | NQA | MVP | TB | SSv2 | VG-T | VG-V | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SigLIP caps+QA | |||||||||||||
| 2k | 51.6 | 59.2 | 49.2 | 59.6 | 23.6 | 44.0 | 66.8 | 12.8 | 52.2 | 67.3 | 34.8 | 23.2 | 45.36 |
| 4k | 55.5 | 62.8 | 50.4 | 62.6 | 25.9 | 46.3 | 68.6 | 30.4 | 54.6 | 71.6 | 48.6 | 23.6 | 50.07 |
| 6k | 59.9 | 65.5 | 56.7 | 63.3 | 26.9 | 47.5 | 70.6 | 36.1 | 55.2 | 73.0 | 50.0 | 25.0 | 52.48 |
| 8k | 62.7 | 65.8 | 58.8 | 63.1 | 26.9 | 48.1 | 71.7 | 37.7 | 58.1 | 74.8 | 49.0 | 24.8 | 53.46 |
| 12k | 64.5 | 68.2 | 60.1 | 65.3 | 29.5 | 48.7 | 72.8 | 44.9 | 57.9 | 79.5 | 50.6 | 25.6 | 55.63 |
| Comparison | Steps | MVB | PT | TVB | TC | TOM | MoB | NQA | MVP | SSv2 | VG-T | all items |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SigLIP+CF V-JEPA+CF | 12–24k | -1.7 | -2.4 | -3.9 | -5.2 | -13.0 | -3.3 | +1.7 | -1.6 | -6.5 | -5.7 | -2.37 |
| SigLIP+CF SigLIP caps | 12–24k | -0.4 | +0.2 | +0.9 | +2.2 | +6.4 | +0.2 | +0.2 | +0.2 | +1.0 | +0.5 | +0.63 |
| SigLIP caps V-JEPA+CF | 12–24k | -1.3 | -2.6 | -4.8 | -7.4 | -19.4 | -3.5 | +1.5 | -1.8 | -7.6 | -6.2 | -3.00 |
| V-JEPA caps V-JEPA+CF | 2–8k | +2.4 | -0.5 | +0.7 | -4.3 | -12.6 | +1.0 | +1.2 | -1.6 | +1.9 | -4.7 | -0.22 |
| SigLIP+CF V-JEPA+CF | 2–8k | -1.3 | -2.1 | -5.3 | -2.0 | -11.6 | -2.7 | +3.1 | -3.6 | -8.5 | -3.9 | -1.97 |
| SigLIP caps V-JEPA caps | 2–8k | -2.4 | -1.7 | -7.0 | +0.6 | -3.4 | -1.5 | +1.7 | -2.6 | -11.1 | -0.5 | -1.89 |
| Comparison | speed | direction | order | camera | count | state | localiz. | antonym | rotation | tracking | TB | SSv2 | NQA | knowl. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CF on V-JEPA, 8k | +32.8 | +3.7 | +1.9 | +6.8 | -1.5 | +4.1 | -5.0 | +3.8 | +2.8 | -8.9 | +2.6 | +0.3 | -0.4 | -0.4 |
| CF on V-JEPA, final | +38.1 | +4.0 | +1.9 | +3.6 | +1.6 | +1.5 | -3.1 | -1.0 | +2.1 | -4.0 | +0.2 | +1.3 | -0.4 | +0.0 |
| AoT only on V-JEPA, 8k | +0.6 | -3.9 | -0.4 | +3.4 | -6.5 | -0.4 | -4.7 | +1.2 | +1.7 | -3.3 | +2.8 | -2.6 | -0.9 | -0.5 |
| CF on SigLIP, 8k | +11.4 | +0.1 | +1.7 | +0.0 | +1.2 | -0.4 | -0.6 | +7.1 | +3.8 | -8.0 | +0.5 | +3.9 | +0.7 | -0.8 |
| CF on SigLIP, 24k | +18.0 | +3.0 | +3.3 | +1.9 | +1.4 | -0.3 | +1.7 | +5.4 | -0.3 | +3.8 | +2.4 | +0.4 | +0.1 | -0.3 |
| CF on SigLIP, final | +16.9 | +3.2 | +3.4 | +1.2 | +2.3 | +1.5 | +2.5 | +5.2 | -1.0 | +0.2 | +1.7 | -0.1 | +0.4 | -0.4 |
| reversal (1,350 items, 5 benchmarks) | mirror (811 items) | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | orig | follows | both | stuck | orig | follows | both | stuck |
| Molmo2-4B | 59.9 | 51.5 | 31.9 | 41.6 | 48.7 | 44.6 | 23.9 | 45.5 |
| InternVL3.5-4B | 54.6 | 48.5 | 29.7 | 46.4 | 53.3 | 51.4 | 35.4 | 50.7 |
| Qwen3-VL-4B | 55.0 | 47.9 | 25.3 | 46.1 | 42.4 | 38.1 | 17.5 | 50.9 |
| PLM-3B | 47.4 | 46.3 | 21.0 | 47.3 | 39.6 | 39.8 | 20.3 | 46.9 |
| V-JEPA caps+QA (2B, 8k) | 56.6 | 47.9 | 32.3 | 40.1 | 50.7 | 50.6 | 34.5 | 48.1 |
| Pair set: share / stuck / leak | SigLIP +CF | V-JEPA caps+QA | V-JEPA +CF |
|---|---|---|---|
| Unseen speed ratios | .065 / 81.7% / 44.8 | .022 / 89.4% / 52.7 | .983 / 1.1% / 0.08 |
| Speed, new source (Kinetics-700) | .611 / 73.2% / 1.24 | .081 / 90.8% / 5.3 | .943 / 6.3% / 0.29 |
| Accel./decel. (+reversal) | .482 / 33.1% / 2.0 | .081 / 79.5% / 9.1 | .745 / 5.4% / 0.83 |
| Direction combinations | .918 / 51.0% / .087 | .900 / 9.2% / .095 | .980 / 2.7% / .021 |
| / proto 8 at layer 18 | |||
| Unseen speed ratios | .008 / 50.1 | .035 / 55.2 | .648 / 99.3 |
| held-out speed | held-out direction | natural, dynamics | natural, content | ||||||||||
| Model | Removed subspace | dims | energy | ratio | dom. | comb. | dom. | comb. | camera | TOM | TC | NQA | PT |
| Ours-2B-Q | none | 0 | 0 | 98.7 | 98.5 | 98.9 | 75.7 | 98.5 | 99.8 | 50.9 | 73.5 | 74.0 | 75.3 |
| speed | 8 | 16.6 | 91.5 | 77.8 | 65.7 | 75.2 | 98.5 | 98.8 | 46.4 | 70.1 | 73.2 | 73.8 | |
| random, energy-matched | 24 | 16.2 | 96.7 | 95.5 | 97.2 | 74.8 | 98.1 | 99.2 | 49.9 | 71.7 | 73.2 | 74.3 | |
| reversal | 8 | 6.0 | 96.9 | 97.8 | 98.9 | 72.4 | 97.9 | 99.4 | 50.3 | 71.6 | 74.0 | 75.2 | |
| mirror | 8 | 3.0 | 98.7 | 98.7 | 99.4 | 74.4 | 97.6 | 99.6 | 50.0 | 74.2 | 74.1 | 75.2 | |
Video-Language Models
Video-language models connect visual content over time with natural language for understanding, describing, retrieving, and interacting with videos.
See all 176 papers