World-model video encoders such as V-JEPA learn how scenes change (direction, speed, order, state) from video alone. To answer questions, they are joined to a language model trained on captions and question-answer pairs. But each caption or question names only a few attributes of a clip, so the model learns to retrieve those attributes rather than to read the clip, and the dynamics the encoder carries go unused: on the same items, a linear probe reads speed from frozen V-JEPA features at 87% while the VLM answers at chance. We show where dynamics lives in a frozen encoder: in a subspace shared across videos for predictive encoders such as V-JEPA, but not as a shared component of frame-wise, text-aligned encoders such as SigLIP. We then build counterfactual supervision that makes the language model read it: pixel transformations turn public videos into answer-balanced variants whose answers are set by the transformation and differ only in the change. Trained in a single stage on 4.53M public samples with 32 visual tokens per frame, where open video LLMs use several stages, 9M to 172M samples and 64 to 256 tokens per frame on average, the resulting V-JEPA-LM models (1B to 8B, Qwen3 and Llama) lead every size class in average score across eleven video benchmarks; our 4B model scores 72.1, against 61.1 for the best open 4B and 64.8 for the best open 8B model. A controlled encoder x supervision study shows that the encoder decides what can be read and the supervision what is read: at a matched budget, counterfactual supervision adds 4.5 points on average and 38 on the speed questions of natural benchmarks. Code and models: https://github.com/kkt94/Llama-V-JEPA-LM.
Figures & tables
Figure 1: Top: what captions and QA teach (left), where the dynamics lives in the encoder (middle), and the supervision built from it (right). Bottom: the model, trained in one stage, and what the recipe changes (before → after: the V-JEPA 2 video LLM [ 4 ] ; accuracy: the best open 4B model).
Figure 2: One generated pair: the same clip forward and backward, with the same question and opposite answers (4 of 32 frames). Other constructions: Fig. 6 .
Family
Construction of Tv
Answer rule πT
Source videos
Rows
Pair%
Sign flip
State change played backwards
action ↔ inverse (open/close)
Charades-Ego, EPIC, SSv2
49k
3
Object direction, clip mirrored or reversed
direction remapped
SSv2
37k
0
Camera pan, tilt or zoom on static footage, or reversed
move remapped
CLEVRER, Charades-Ego, SSv2
23k
0
Two clips joined, each optionally reversed
direction A then B
SSv2
7k
0
Order
Two events in sequence or stacked at once
first / second / simultaneous
LLaVA-Video, SA-V, SSv2
65k
0
Segment matching a two-action caption
matching segment
HoloAssist
20k
80
Table 1: The fourteen counterfactual constructions (33 sources, 509k rows, 6.1% of sampled training rows). Each row is rendered at training time from a clip of a public dataset and a transformation; the answer follows from πT . Pair%: rows whose counterpart from the same clip with a different answer is also in the same source. Per-source details, real questions and rendered examples: Appendix B .
reversal
speed
mirror
speed readout
Encoder
rd
κ
κ
κ
proto 8
probe
SigLIP so400m
0
–
.0016
.0014
50.0
73.5
ViCLIP (text-aligned)
.052
.0012
.042
.0019
60.1
70.3
VideoMAE (pixel pred.)
.502
.0032
.632
.045
100.0
76.8
V-JEPA ViT-L (1B, 2B)
.064
.037
.080
.031
66.3
81.2
V-JEPA ViT-g (3B–8B)
.108
.039
.167
.074
77.2
84.7
Table 2: Change component of frozen encoders (1,200 clips; 70% fit, 30% held out). rd : relative size of the in-token change after aligning token positions; κ : cross-video consistency; proto 8: accuracy of a prototype read from 8 pairs (chance 50); probe: linear probe on all training clips. ViCLIP and VideoMAE: diagnostic encoders with joint space–time attention, not used in our models. All measures: Tab. 22 .
linear probe on the stage
VLM
Encoder
Training
frozen
bridge
L4
L14
L18
L28
answer
SigLIP
caps+QA
36.4
38.7
36.8
35.9
35.7
34.7
33.1
SigLIP
+CF
36.3
39.1
37.5
37.1
35.9
35.8
34.4
V-JEPA
caps+QA
86.7
83.5
53.8
47.2
49.4
46.9
33.7
V-JEPA
+CF
86.8
91.2
66.6
78.9
96.3
96.5
92.8
Table 3: One item set at every stage: the 540 held-out speed-ratio items (unseen ratios 1.6 / 2.5 / 4.0 and equal; SSv2 validation; 3 classes, chance 33.3), 2B models at 8k steps. Probe: linear (PCA 64, logistic) at the answer position for language-model layers and on a 16×2×2 token grid for the frozen encoder and the bridge, 5-fold cross-validation grouped by source clip, 4 probe seeds (s.d. ≤1.8 ). Answer: the VLM on the same inputs (one option order per clip), within 2 points of the evaluation in Tab. 6 .
Model
MVB
PT
TVB
TC
TOM
MoB
NQA
MVP
TB
SSv2
VG-T
VG-V
Avg
2B class
Qwen3-VL-2B
61.0
65.1
49.8
67.4
30.3
57.5
77.1
32.3
65.9
59.2
36.2
27.6
52.45
InternVL3.5-2B
66.8
64.2
50.9
65.7
31.0
56.2
76.1
28.8
71.3
57.0
40.0
21.0
52.42
PLM-1B
68.0
69.8
48.2
60.5
24.5
51.7
79.8
30.9
58.1
61.7
33.2
29.6
51.33
Apollo-1.5B
61.1
63.9
47.9
60.3
27.8
51.1
78.8
26.0
55.8
56.0
38.8
22.4
49.16
VideoLLaMA3-2B
57.9
62.9
44.0
64.1
24.0
56.3
75.4
23.1
64.1
50.0
38.0
25.2
48.75
Table 4: Main results: twelve scores on eleven video benchmarks. All models evaluated by us with one protocol (32 frames, same prompts, option-letter scoring). Shaded: V-JEPA-LM (-Q: Qwen3, -L: Llama). Bold: best in class. MVB: MVBench, PT: PerceptionTest, TVB: TVBench, TC: TempCompass, TOM: TOMATO, MoB: MotionBench, NQA: NExT-QA, MVP and VG-T/VG-V (Vinoground text/video): pair accuracy, TB: TemporalBench (binary), SSv2: Something-Something v2 (multiple choice). 95% bootstrap intervals of the average: ± 0.7 for every model ( Appendix C ). Gemma 4, last in both classes: Tab. 16 .
Model
Tok.
Samples
MVB
PT
TVB
TC
TOM
MVP
Avg
V-JEPA 2 (L3.1-8B)
9,216
88.5M
73.5
84.0
60.6
76.9
40.3
44.5
63.3
V-JEPA 2.1 (L3.1-8B)
–
72.5M
75.4
83.1
62.8
74.0
38.0
43.2
62.8
Ours-8B-Q (Qwen3-8B)
1,024
4.53M
72.8
77.8
75.3
79.4
56.5
58.1
70.0
Ours-8B-L (L3.1-8B)
1,024
4.53M
71.9
75.0
71.8
77.6
55.5
57.6
68.2
Table 5: Against Meta’s V-JEPA video LLMs on the six benchmarks both report [ 4 , 46 ] (L3.1-8B: Llama-3.1-8B, also the language model of Ours-8B-L). Meta values quoted from the papers (the video LLMs are not released, only the encoders): V-JEPA 2 with ViT-g 384 , V-JEPA 2.1 with ViT-G 384 ; PerceptionTest is the test split after training on its train split for V-JEPA 2 and the validation split for V-JEPA 2.1. Ours measured.
Figure 3: Twelve-score average against training samples as reported (log scale). Marker area: visual tokens per clip. Sample counts: Appendix C .
Figure 4: Twelve-score average during training (2B). Solid: with counterfactual rows; dashed: captions and QA only.
12-score avg
abilities at 8k (natural benchmarks)
held-out sets at 8k
Encoder
Supervision
8k
final
speed
direction
camera
knowledge
6-set avg
unseen ratios
rev. cons.
SigLIP so400m
caps+QA
53.46
58.52
34.0
39.9
36.4
63.5
47.1
35.0
19.4
SigLIP so400m
+CF
56.77
62.13
45.4
40.0
36.4
62.6
59.3
34.8
19.7
V-JEPA ViT-L
caps+QA
56.65
61.53
32.4
59.7
40.5
64.1
58.6
33.5
32.3
V-JEPA ViT-L
AoT only
56.37
61.17
33.0
55.7
43.9
63.6
60.8
31.9
30.8
V-JEPA ViT-L
+CF
61.18
66.02
65.3
63.3
47.4
63.7
91.3
91.5
36.8
Table 6: Encoder × supervision (2B; same mixture and schedule). +CF: with the counterfactual rows; AoT only: an arrow-of-time question in their place. Final: step 27,382. Abilities: Sec. 6.1 ; held-out chance 33.3; rev. cons.: original and reversed video both correct ( Tab. 27 ). All steps and benchmarks: Tab. 24 .
new
new corpus
new comb.
new
succ.
Model
ratio
spd
dir
dir
spd
cam.
avg
play
Qwen3-VL-2B
37.0
45.5
35.2
56.5
44.6
43.9
43.8
33.1
Qwen3-VL-4B
35.9
50.5
40.7
73.3
50.2
49.4
50.0
36.9
Qwen3-VL-8B
36.7
50.8
41.9
78.4
68.0
57.1
55.5
51.9
V-JEPA, no CF
33.3
44.3
69.8
97.1
49.1
92.4
64.3
39.4
Ours-2B-Q
98.7
98.5
75.7
98.5
98.9
99.8
95.0
93.9
Table 7: Generalization (chance 33.3): the six held-out sets of Sec. 5.2 and successive playback (succ. play). V-JEPA, no CF: the 2B captions and QA arm of Tab. 6 ; like the open models, it never saw these constructions. All models: Tabs. 18 and 20 .
Figure 5: Gain from counterfactual supervision per ability (2B, 8k steps); whiskers: 95% paired bootstrap interval.
held-out speed
content
Removed subspace
energy
ratio
comb.
TOM
NQA
PT
none
0
98.7
98.9
50.9
74.0
75.3
speed directions (8)
16.6
91.5
65.7
46.4
73.2
73.8
random, same energy (24)
16.2
96.7
97.2
49.9
73.2
74.3
top content directions (24)
43.6
95.0
84.3
44.8
72.4
72.0
Table 8: Removing a subspace from every visual token of Ours-2B-Q at inference. Energy: % removed. Full: Tab. 29 .
egocentric actions with reversible verbs, clip reversed
16,780
0.26
7
Which action does the person perform?
Appendix
Table 10: All 33 counterfactual sources of the training mixture (509,193 rows, 6.11% of sampled rows). Paired %: share of a source’s rows whose counterpart from the same clip with a different answer is also in that source (29.4% overall); the other rows are single members of answer-balanced constructions ( Sec. 4 ).
Figure 6: One training row per construction (4 of 32 frames, after the transformation), with its question and answer.
Model
Language model
Encoder
Batch/GPU × GPUs
Global
Steps
Samples
LR (LLM)
EFLOPs
Ours-2B-Q
Qwen3-1.7B
ViT-L
12 × 14
168
27,382
4.60M
1.05e-5
121
Ours-1B-L
Llama-3.2-1B-Instr.
ViT-L
6 × 28
168
27,382
4.60M
1.05e-5
104
Ours-4B-Q
Qwen3-4B-Instr.-2507
ViT-g
5 × 30
150
30,187
4.53M
1.2e-5
314
Ours-3B-L
Llama-3.2-3B-Instr.
ViT-g
10 × 30
300
15,094
4.53M
1.7e-5
277
Ours-8B-Q
Qwen3-8B
ViT-g
5 × 30
150
30,187
4.53M
8e-6
437
Ours-8B-L
Llama-3.1-8B-Instr.
ViT-g
5 × 30
150
30,187
4.53M
8e-6
426
Appendix
Table 11: Training settings. All: frozen V-JEPA 2.1 encoder at 32 frames, spectral bridge to 1,024 tokens (fixed PCA rotation and rescaling, then a trainable MLP), bridge learning rate 2×10−4 (3B: 2.83×10−4 ), weight decay 0.01 (LLM) / 0.05 (bridge), linear warm-up over 0.2–0.3% of steps with the LLM frozen for the first 0.6–0.8%, cosine decay, maximum sequence 2,560 tokens, 4.53M rows drawn at fixed per-source weights from the 175-source mixture (4.60M for the ViT-L models). Compute: Tab. 12 ; EFLOPs ( 1018 ): trained parameters × trained tokens × 6 plus attention and the frozen encoder’s forward pass, measured per clip (activation recomputation excluded).
Figure 7: Training loss, gradient norm (bin median) and language-model learning rate of the released models against samples seen.
Model
H100s
Global batch
Steps
Samples
Samples/s
Hours
H100-hours
Peak mem. (GB)
Loss, first 1%
Loss, last 5%
Ours-2B-Q
14
168
27,382
4.60M
78
16.4
229
46
1.53
0.41
Ours-1B-L
28
168
27,382
4.60M
118
10.8
303
18
1.31
0.43
Ours-4B-Q
30
150
30,187
4.53M
76
16.6
499
44
1.41
0.39
Ours-3B-L
30
300
15,094
4.53M
88
14.3
429
51
1.51
0.39
Ours-8B-Q
30
150
30,187
4.53M
58
21.6
648
33
1.36
0.38
Ours-8B-L
30
150
30,187
4.53M
63
19.9
596
32
1.37
0.40
Appendix
Table 12: Training metrics of the released models (one stage, one run each). Samples/s: median throughput over the run; hours and H100-hours: samples divided by that throughput, so pauses between resumed segments are not counted. Peak memory: per GPU (activation checkpointing). Loss: mean next-token cross-entropy on answer tokens of the logged batches.
Samples (M)
2.7
3.0
3.3
3.6
3.9
4.2
4.5
4.53
Ours-4B-Q
69.97
70.24
69.98
71.43
71.41
71.73
72.18
72.14
Ours-3B-L
66.40
67.14
67.39
68.33
68.73
68.54
68.92
68.97
Ours-8B-Q
68.66
68.47
69.28
69.89
70.12
69.92
70.23
70.11
Ours-8B-L
68.40
68.82
68.79
69.78
70.41
70.59
70.61
70.51
Appendix
Table 13: Twelve-score average at matched training samples for the ViT-g models (3B: global batch 300; 4B, 8B: 150). Each model is first in its class at every point. The ViT-L models (1B, 2B; global batch 168) are checkpointed at other sample counts; the 2B trajectory is in Fig. 4 .
Model
MVBench
PerceptionTest
TVBench
TempCompass
TOMATO
MotionBench
NExT-QA
2B class
Qwen3-VL-2B
61.7 (61.0)
– (65.1)
– (49.8)
– (67.4)
– (30.3)
– (57.5)
– (77.1)
InternVL3.5-2B
65.9 (66.8)
– (64.2)
– (50.9)
– (65.7)
– (31.0)
– (56.2)
– (76.1)
PLM-1B
70.1 (68.0)
72.7 t (69.8)
50.4 (48.2)
64.6 (60.5)
25.5 (24.5)
52.2 (51.7)
80.3 (79.8)
VideoLLaMA3-2B
65.5 (57.9)
68.0 t (62.9)
– (44.0)
63.4 (64.1)
– (24.0)
– (56.3)
81.1 (75.4)
Apollo-1.5B
– (61.1)
61.0 (63.9)
– (47.9)
60.8 (60.3)
– (27.8)
– (51.1)
– (78.8)
Appendix
Table 14: Scores reported by the models’ authors, with ours under the common protocol in parentheses (32 frames, option-letter scoring; Tab. 4 ). Reported settings differ from ours: PerceptionTest test split ( t ; ours: validation), Qwen3-VL at 2 fps up to 2,048 frames, InternVL3.5 the best of 16–64 frames, Molmo2 at 2 fps up to 128 frames, VideoLLaMA3 at 1 fps up to 180 frames, PLM 32 frames; PLM’s TVBench from its repository. –: not reported. Gemma 4 reports no video benchmark; MVP, TemporalBench (binary), SSv2 and Vinoground are not reported by any of these models.
Benchmark
Items
Shared
Shared source videos
MVBench
4,000
60 (1.5%)
Charades (STAR) clips also in LLaVA-Video-178K
TVBench
2,525
57 (2.3%)
the same Charades clips; PerceptionTest train
TemporalBench
2,765
30 (1.1%)
YouTube videos also in Kinetics-700
The other nine sets
26,198
0
–
All twelve
35,488
147 (0.41%)
Appendix
Table 15: Video-level audit of all 175 training sources against every evaluation item. Rescoring without the shared items moves each model by at most 0.42 points on a benchmark and 0.04 on the twelve-score average, for ours and the open models alike.
Model
MVB
PT
TVB
TC
TOM
MoB
NQA
MVP
TB
SSv2
VG-T
VG-V
Avg
Gemma 4 E2B
39.8
46.3
36.4
51.5
18.7
41.1
51.4
8.1
61.4
37.0
24.0
22.4
36.51
Gemma 4 E4B
46.5
52.0
41.5
57.4
22.8
49.4
64.7
16.5
61.0
43.8
26.4
24.0
42.17
Appendix
Table 16: Gemma 4 under the common protocol (thinking off, its own processor and chat template). Its pair-accuracy scores fall far below those of the other open models of its size (MVP 8.1 and 16.5), so it is listed here rather than in Tab. 4 ; it would rank last in its class either way.
Set
Items
Source
What is new
Speed ratio
540
SSv2 validation
relative speed of a clip stacked over itself at span ratios 1.6, 2.5, 4.0 (training: equal, 2:1 and 3:1, and 1.0–1.4 in the fine-motion source; none of the three ratios occurs), plus the equal item of each clip
Domain, speed
600
Kinetics-700 validation
relative speed (2:1, 3:1, equal) and acceleration on a corpus never used by the speed family
Domain, direction
540
Kinetics-700 validation
camera direction labeled by optical flow, original / reversed / mirrored, on a corpus unused for this family
Combo, direction
584
SSv2 validation
mirror and reversal together (the answer is the original direction), with the three trained variants of the same clip
Combo, speed
540
SSv2 validation
acceleration played backwards (accelerating becomes decelerating), plus freeze
Camera, domain
490
NVIDIA PhysX simulation
synthetic camera moves on a simulator never used for camera rows
Appendix
Table 17: Held-out generalization sets (chance 33.3 on average). Question wordings, option pools and option-length balancing are the training builders’ own.
Model
ratio
dom. spd
dom. dir
comb. dir
comb. spd
camera
avg
Open models
Qwen3-VL-2B
37.0
45.5
35.2
56.5
44.6
43.9
43.8
Qwen3-VL-4B
35.9
50.5
40.7
73.3
50.2
49.4
50.0
InternVL3.5-4B
33.9
46.7
30.7
61.1
53.0
45.1
45.1
Molmo2-4B
38.3
50.5
29.1
59.4
51.1
32.7
43.5
PLM-3B
35.9
49.8
35.7
64.9
58.5
44.9
48.3
Appendix
Table 18: Held-out construction sets (definitions in Tab. 17 : unseen speed ratios, an unseen source for the speed and direction families, unseen combinations, a simulator unused for camera rows). Ratio: unseen playback-speed ratios (chance 33.3); dom.: speed / direction on a new source corpus; comb.: unseen mirror × reversal and acceleration × reversal combinations; camera: simulated camera motion. 4B-class open models average 43.5–50.0 and Qwen3-VL-2B 43.8.
inside training range
beyond
Model
1.6
2.5
4.0
Controlled 2B arms, 8k
SigLIP +CF
38.3
35.6
30.6
V-JEPA caps+QA
32.8
34.4
33.3
V-JEPA temporal QA in place of CF
37.8
36.7
33.9
V-JEPA +CF
88.9
91.7
93.9
Appendix
Table 19: Unseen speed ratios separately (180 items each, chance 33.3). Training uses equal, 2:1 and 3:1 and fine ratios up to 1.4, so 1.6 and 2.5 lie between trained ratios and 4.0 beyond them.
Model
Single
Pair
Open models
Qwen3-VL-2B
33.1
8.9
Qwen3-VL-4B
36.9
1.1
InternVL3.5-4B
24.2
0.6
Molmo2-4B
19.4
4.4
PLM-3B
29.2
2.8
Appendix
Table 20: Successive playback (360 items, 180 clips): the speed-ratio set’s clips and unseen ratios, the clip played twice one after the other, asked whether the motion speeds up or slows down. Single: chance 33.3; pair: both orders of a clip correct, chance 11.1.
vision encoder
connector
LLM prefill
first token
decode
720p clip (ms)
Model
vis. tokens
TFLOPs
ms
ms
TFLOPs
ms
ms
ms/token
LLM prefill
first token
2B class
Qwen3-VL-2B
1,920 / 11,520
5.0
35
in enc.
7.0
31
66
18.2
154
308
InternVL3.5-2B
8,192
23.2
55
0.4
40.5
104
159
18.4
104
159
PLM-1B
8,192
22.2
87
1.8
25.2
57
146
9.3
57
146
Ours-2B-Q
1,024
13.9
60
0.8
3.3
22
83
20.0
21
82
Appendix
Table 21: Inference cost per 32-frame clip on one H100 (batch 1, bf16, video decoding excluded; times: median of 10 runs after 3 warm-ups; 480p MVBench clip unless noted). TFLOPs: matrix multiplications including attention, counted with PyTorch’s FLOP counter. Vision encoder: the vision tower; for our models the frozen, unmodified V-JEPA 2.1 forward (ViT-L for 1B and 2B, ViT-g otherwise), a cost set by the choice of encoder and identical for any recipe built on it: V-JEPA attends jointly over all frames, while the open models’ encoders attend within a frame or frame pair. Connector: projector or bridge (“in enc.”: part of the vision module; at most 1.5 TFLOPs for every model). LLM prefill: the language model over the whole prompt; first token: all three parts; decode: per generated token with KV cache (128 tokens). Visual tokens: 480p / 720p where they differ (Molmo2: from its processor). Our token count and cost do not depend on resolution; Qwen3-VL’s grow with it. –: not timed.
Encoder
Transf.
rd
κ
ρ
proto 8
proto all
SigLIP so400m
reversal
0
–
–
50.0
50.0
speed
.016
.0016
.18
50.0
50.3
mirror
.066
.0014
.18
49.8
49.0
V-JEPA ViT-L
reversal
.064
.037
.12
58.8
63.7
speed
.055
.080
.40
66.3
71.0
mirror
.093
.031
.031
58.7
70.4
Appendix
Table 22: Encoder decomposition on 1,200 clips (token-averaged d ; 70% fit, 30% held out). κ uses y=+1 for every reversal and speed pair (the direction of the transformation) and y=±1 for left-to-right / right-to-left motion under mirroring, on frozen features pooled to 4×4 cells per tubelet and standardised per dimension on the fitting clips; random directions give κ≈1/840≈0.001 . ρ : overlap of the change direction with the 32 leading content components (a content direction scores 0.44–0.63). SigLIP’s reversal component is zero by construction. ViCLIP [ 63 ] : video–text contrastive training with joint attention over all frame tokens (16 frames, the 8-frame temporal embedding interpolated); VideoMAE [ 58 ] : masked pixel prediction with 2×16×16 tubelets (32 frames); both at 2242 (ViCLIP: penultimate block; VideoMAE: last layer), on the same clips, variants and pooling.
κ (frozen)
CF gain at 8k
Family
Transformation
Theory
Prediction
V-JEPA
SigLIP
V-JEPA
SigLIP
Ability
Speed, rate
rescaling τa
Proposition 1 (i); measured κ
large on V-JEPA, small on SigLIP
.080
.0016
+32.8
+11.4
speed
Sign flips
reversal, camera moves
Proposition 2 : d∈V− ; Proposition 1 (i): d≡0
large on V-JEPA
.037
–
+6.8
0.0
camera
+4.1
−0.4
state change
Sign flips
mirror
Proposition 2 ; Proposition 1 (ii)
large on V-JEPA
.031
.0014
+3.7
+0.1
direction
Event order
segment permutation
local receptive field: A=I
equal on both
–
–
+1.9
+1.7
order
Appendix
Table 23: From encoder structure to gain. For each family: the transformation, the result that predicts how its change is carried, the prediction, the measured cross-video consistency κ of the frozen ViT-L and SigLIP ( Tab. 22 ; SigLIP reversal has rd=0 , no in-token component), and the CF gain on the matching natural-benchmark ability at 8k steps ( Tab. 26 ). Order changes token positions rather than token content, so both encoders gain alike. Speed is the one family the propositions do not cover for V-JEPA (a rate change gives no shared direction by Proposition 2 , Appendix A ); its prediction rests on the measured κ alone, with Proposition 1 (i) giving SigLIP only scene-specific frame differences. Gains are those of the whole counterfactual mixture against captions and QA, read per ability; at the final step the direction and state-change gains of the two encoders draw closer, while speed stays far apart ( Tab. 26 ).
Step
MVB
PT
TVB
TC
TOM
MoB
NQA
MVP
TB
SSv2
VG-T
VG-V
Avg
SigLIP caps+QA
2k
51.6
59.2
49.2
59.6
23.6
44.0
66.8
12.8
52.2
67.3
34.8
23.2
45.36
4k
55.5
62.8
50.4
62.6
25.9
46.3
68.6
30.4
54.6
71.6
48.6
23.6
50.07
6k
59.9
65.5
56.7
63.3
26.9
47.5
70.6
36.1
55.2
73.0
50.0
25.0
52.48
8k
62.7
65.8
58.8
63.1
26.9
48.1
71.7
37.7
58.1
74.8
49.0
24.8
53.46
12k
64.5
68.2
60.1
65.3
29.5
48.7
72.8
44.9
57.9
79.5
50.6
25.6
55.63
Appendix
Table 24: Encoder × supervision, every evaluated checkpoint (2B, global batch 168). The V-JEPA caps+QA arm was trained to 8k first and then continued to the final step; the AoT-only arm likewise, and the temporal-QA arm was trained to 8k. Temporal QA: the counterfactual rows removed and their weight given to the 22 annotation-based temporal-QA sources of the mixture (temporal grounding, highlights, step order, before/after, next action, repetition counting, unintentional action).
Comparison
Steps
MVB
PT
TVB
TC
TOM
MoB
NQA
MVP
SSv2
VG-T
all items
SigLIP+CF − V-JEPA+CF
12–24k
-1.7
-2.4
-3.9
-5.2
-13.0
-3.3
+1.7
-1.6
-6.5
-5.7
-2.37
SigLIP+CF − SigLIP caps
12–24k
-0.4
+0.2
+0.9
+2.2
+6.4
+0.2
+0.2
+0.2
+1.0
+0.5
+0.63
SigLIP caps − V-JEPA+CF
12–24k
-1.3
-2.6
-4.8
-7.4
-19.4
-3.5
+1.5
-1.8
-7.6
-6.2
-3.00
V-JEPA caps − V-JEPA+CF
2–8k
+2.4
-0.5
+0.7
-4.3
-12.6
+1.0
+1.2
-1.6
+1.9
-4.7
-0.22
SigLIP+CF − V-JEPA+CF
2–8k
-1.3
-2.1
-5.3
-2.0
-11.6
-2.7
+3.1
-3.6
-8.5
-3.9
-1.97
SigLIP caps − V-JEPA caps
2–8k
-2.4
-1.7
-7.0
+0.6
-3.4
-1.5
+1.7
-2.6
-11.1
-0.5
-1.89
Appendix
Table 25: Paired item-level differences pooled over four checkpoints (each item’s mean difference over the steps; bootstrap over items). +CF: with counterfactual supervision; caps: caption and QA mixture alone. 95% intervals exclude zero for every entry with ∣Δ∣≥1.6 (e.g. TOMATO −13.0[−15.2,−10.8] , SSv2 −6.5[−7.7,−5.4] , NExT-QA +1.7[+1.0,+2.3] in the first row). “All items” weights benchmarks by size (NExT-QA has 8.6k items).
Comparison
speed
direction
order
camera
count
state
localiz.
antonym
rotation
tracking
TB
SSv2
NQA
knowl.
CF on V-JEPA, 8k
+32.8
+3.7
+1.9
+6.8
-1.5
+4.1
-5.0
+3.8
+2.8
-8.9
+2.6
+0.3
-0.4
-0.4
CF on V-JEPA, final
+38.1
+4.0
+1.9
+3.6
+1.6
+1.5
-3.1
-1.0
+2.1
-4.0
+0.2
+1.3
-0.4
+0.0
AoT only on V-JEPA, 8k
+0.6
-3.9
-0.4
+3.4
-6.5
-0.4
-4.7
+1.2
+1.7
-3.3
+2.8
-2.6
-0.9
-0.5
CF on SigLIP, 8k
+11.4
+0.1
+1.7
+0.0
+1.2
-0.4
-0.6
+7.1
+3.8
-8.0
+0.5
+3.9
+0.7
-0.8
CF on SigLIP, 24k
+18.0
+3.0
+3.3
+1.9
+1.4
-0.3
+1.7
+5.4
-0.3
+3.8
+2.4
+0.4
+0.1
-0.3
CF on SigLIP, final
+16.9
+3.2
+3.4
+1.2
+2.3
+1.5
+2.5
+5.2
-1.0
+0.2
+1.7
-0.1
+0.4
-0.4
Appendix
Table 26: Ability groups over the natural benchmarks’ own categories (2B; final: step 27,382). Items: speed 527, direction 1,170, order 1,457, camera 585, count 1,776, state change 711, localization 360, reversal/antonym 520, rotation 286, tracking 425, TemporalBench 2,765, SSv2 2,088, NExT-QA 8,564, knowledge (objects, scenes, knowledge) 9,254. Paired bootstrap 95% intervals: speed on V-JEPA [+27.9,+37.8] , on SigLIP [+6.3,+16.5] (8k) and [+12.7,+23.3] (24k); direction between encoders without CF [+16.7,+22.7] ; speed between encoders without CF [−4.9,+1.7] .
reversal (1,350 items, 5 benchmarks)
mirror (811 items)
Model
orig
follows
both
stuck
orig
follows
both
stuck
Molmo2-4B
59.9
51.5
31.9
41.6
48.7
44.6
23.9
45.5
InternVL3.5-4B
54.6
48.5
29.7
46.4
53.3
51.4
35.4
50.7
Qwen3-VL-4B
55.0
47.9
25.3
46.1
42.4
38.1
17.5
50.9
PLM-3B
47.4
46.3
21.0
47.3
39.6
39.8
20.3
46.9
V-JEPA caps+QA (2B, 8k)
56.6
47.9
32.3
40.1
50.7
50.6
34.5
48.1
Appendix
Table 27: Counterfactual consistency in full. Items qualify automatically when the correct option, with the transformation’s word swaps applied (left ↔ right, towards ↔ away, open ↔ close, before ↔ after, …), equals another option exactly. Orig: accuracy on the original video; follows: the transformed answer is the swapped option; both: both correct; stuck: same answer for both videos.
Pair set: share / stuck / leak
SigLIP +CF
V-JEPA caps+QA
V-JEPA +CF
Unseen speed ratios
.065 / 81.7% / 44.8
.022 / 89.4% / 52.7
.983 / 1.1% / 0.08
Speed, new source (Kinetics-700)
.611 / 73.2% / 1.24
.081 / 90.8% / 5.3
.943 / 6.3% / 0.29
Accel./decel. (+reversal)
.482 / 33.1% / 2.0
.081 / 79.5% / 9.1
.745 / 5.4% / 0.83
Direction combinations
.918 / 51.0% / .087
.900 / 9.2% / .095
.980 / 2.7% / .021
κ / proto 8 at layer 18
Unseen speed ratios
.008 / 50.1
.035 / 55.2
.648 / 99.3
Appendix
Table 28: Internal measurements. Parallelogram: cos(hplain−hmirror,hrev−hmirror+rev) ; −1 means mirror and reversal flip the same subspace ( Proposition 2 ); clips paired with other clips give about 0.03.
held-out speed
held-out direction
natural, dynamics
natural, content
Model
Removed subspace
dims
energy
ratio
dom.
comb.
dom.
comb.
camera
TOM
TC
NQA
PT
Ours-2B-Q
none
0
0
98.7
98.5
98.9
75.7
98.5
99.8
50.9
73.5
74.0
75.3
speed
8
16.6
91.5
77.8
65.7
75.2
98.5
98.8
46.4
70.1
73.2
73.8
random, energy-matched
24
16.2
96.7
95.5
97.2
74.8
98.1
99.2
49.9
71.7
73.2
74.3
reversal
8
6.0
96.9
97.8
98.9
72.4
97.9
99.4
50.3
71.6
74.0
75.2
mirror
8
3.0
98.7
98.7
99.4
74.4
97.6
99.6
50.0
74.2
74.1
75.2
Appendix
Table 29: Removing change subspaces at the bridge input (final 2B checkpoints, no retraining). The subspaces are estimated once in the equalised space of Ours-2B-Q from 400 SSv2 training clips: the top 8 directions of the token-averaged response to playing at twice the speed, to reversal and to mirroring (on the other 20% of clips they capture 46%, 57% and 74% of that response); every visual token is projected off the subspace. Energy: share of token energy removed (%). Energy-matched random: a random 24-dimensional subspace inside a band of content principal components with the same removed energy as the speed subspace. Speed removal takes the unseen speed combinations from 98.9 to 65.7 and TOMATO from 50.9 to 46.4, while the energy-matched random subspace leaves them at 97.2 and 49.9; content benchmarks move by at most 1.5 in both. The model trained without counterfactual rows is at chance on speed with or without the subspace.
Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing self-supervised methods either prevent representation collapse through architectural asymmetries, coupling an exponential-moving-average target encoder, a stop-gradient, and a capacity-limited predictor, or circumvent it by reconstructing masked content in pixel space. We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective, which dispenses with both. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg, which excludes collapse with a provable guarantee. The architecture reduces to an encoder and a projector, and the objective to a single hyperparameter. This formulation admits two properties. First, the cost of pretraining is governed by the number of tokens the encoder observes; uniform random token dropping renders this number small while simultaneously improving downstream accuracy. At matched epochs on identical data, LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute, and at matched total FLOPs it exceeds the strongest video baseline by 7.6 points on ImageNet-1K while remaining competitive on motion-centric benchmarks. Second, since no asymmetry between branches is required, the encoder can be trained with block-causal attention at no measurable accuracy cost: temporal ordering becomes a property of the encoder itself. Against a compute-matched DINOv2 trained on frames of the same videos, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. These results indicate that, once its computational overhead is removed, video becomes a viable and in several respects preferable substrate for general-purpose visual pretraining.
Lukas Kuhn, Lucas Maes, Giuseppe Serra +4
German Cancer Research Center · German Cancer Consortium · Mila +5
Video large language models (Video LLMs) can achieve strong video-QA accuracy without reliably tracking spatiotemporal dynamics. A model may answer a motion question from static cues, for example, and give the same prediction even after the underlying motion is reversed. Correctness-based reinforcement learning does not directly address this problem because it rewards the final answer without requiring the policy to respond to task-relevant video dynamics. We propose Counterfactual Relational Policy Optimization (CRPO), which explicitly trains Video LLMs to respond to controlled changes in the visual input. For each training example, CRPO constructs a counterfactual video, such as a horizontally flipped or temporally reversed version of the original, and jointly optimizes rollouts from both videos under a shared policy. Factual supervision anchors what the model should answer, while counterfactual supervision constrains when that answer should change: predictions should change when an intervention alters task-relevant dynamics and remain stable when the queried property is preserved. This coupling provides a direct behavioral learning signal without requiring ground-truth labels for transformed videos or annotated spatiotemporal reasoning traces, while discouraging indiscriminate answer changes. To evaluate this property, we introduce DyBench, a paired counterfactual benchmark with 3{,}014 videos and a strict pair-accuracy metric. Across paired spatiotemporal evaluations and standard video benchmarks, CRPO improves sensitivity to motion and temporal changes while improving performance on general video understanding. The gains also extend to segment reordering, a transformation never used during training, suggesting that CRPO learns sensitivity to video dynamics beyond the training interventions. The project website can be found at https://ddz16.github.io/crpo.github.io/ .
Dazhao Du, Jian Liu, Jialong Qin +7
Hong Kong University of Science and Technology · Tencent
Streaming video-language models must decide not only what to answer, but whether the evidence needed for the current question has arrived. Existing systems learn that decision as a separate trigger; we ask whether an unmodified model already computes it. We show that frozen VideoLLMs carry a linearly readable evidence-readiness signal, labelled from timestamped evidence rather than from model output. It decodes in all seven models of a shared byte-identical evaluation (AUROC 0.733-0.905 under the strictest not-ready sampling, where a fitted clock is near chance), and a probe fitted without any of a benchmark family's footage still reads that family. It is question-conditioned: on byte-identical windows, changing only the question reverses the readout on 66.1% of pairs, while every question-blind control is at chance by construction. The model can answer incorrectly and still encode readiness: AUROC remains 0.722 among wrong answers. Readiness also beats uncertainty estimators and their supervised combination on latency-matched answer selection, and tracks independent human judgments more closely than confidence. Released streaming triggers are also linear readouts, yet a trained trigger read on its own base model's activations is approximately orthogonal to readiness and decodes it far less accurately than a probe. We turn the readout into Readiness Gating, an answer-timing policy that improves accuracy by up to +9.75 pp at matched video duration with negligible computational overhead. How much it gains varies with the accuracy headroom the task makes available: across 26 configurations the gain tracks that headroom, and an intervention that moves it over identical pixels moves the gain with it.
Dan Ben-Ami, Kobi Cohen, Chaim Baskin
INSIGHT Lab, Ben-Gurion University of the Negev, Israel · Ben-Gurion University of the Negev, Israel