Detecting Temporal Concepts in Video Transformers via a No-False-Positives Test

Jianghuai Li; Andrew Ren; Greg Shakhnarovich

cs.CVOct 23, 2026

Detecting Temporal Concepts in Video Transformers via a No-False-Positives Test

Authors: Jianghuai Li, Andrew Ren, Greg Shakhnarovich

Organizations: University of Chicago · Toyota Technological Institute at Chicago

Abstract

Video models have the architecture to support encoding temporal aspects of the input, i.e. concepts that require more than one frame to define, like object movement. However, because motion and appearance change are entangled, a feature that appears to encode temporal concepts like object movement may instead encode non-temporal concepts like spatial presence at different positions whose activations merely correlate with motion. To resolve this, we introduce the No-False-Positives (NFP) test, a statistical test with a formal guarantee of statistical specificity for identifying features encoding temporal concepts. Using specially designed stimuli, we apply this test to representations produced by VideoMAE, V-JEPA2, and TimeSformer, and observe that temporal features exist in each model. We validate the test first by showing it yields no temporal features for the DINOv2 image-encoder. Then, using a synthetic representation space, we show that detection rate increases with a direction's projection onto the temporal subspace, and that Sparse Autoencoder features test-flagged as temporal project much more of their norm onto the temporal subspace than unflagged features. Interventions on temporal SAE features yield mixed results across models, suggesting that intervening on temporal features may not outperform a non-temporal feature baseline.

Figures & tables

Model	$D$	Features	speed	$v_{x}$	$v_{y}$	$\|a\|$	direction	Any	%
VideoMAE finetuned	768	axes	352	359	151	496	89	707	92.1
VideoMAE finetuned	6144	SAE	66	39	33	19	4	109	1.77
VideoMAE	768	axes	6	23	58	515	29	540	70.3
VideoMAE	6144	SAE	0	0	2	170	0	172	2.80
V-JEPA2	1024	axes	45	2	4	254	0	294	28.7
V-JEPA2	8192	SAE	4	1	2	57	0	62	0.76

Table 1: Final layer representation space dimension axes flagged by the NFP test at the Bonferroni threshold

p<0.05/D

. Axes: Dimension axes of the final layer representation. SAE: sparse autoencoder features.

D

is the number of directions

w

tested (dimensions or SAE dictionary size). Per-profile columns count features significant for that

\tau_{k}

, and a feature can be significant for several. “Any” counts directions

w

significant for at least one profile.

			$\max_{k}\|t\|$ , single-concept directions
$\beta^{2}$	Flagged	Median $\max_{k}\|t\|$	speed	$v_{x}$	$v_{y}$	$\|a\|$	direction
0	0/200	1.5	–	–	–	–	–
$10^{-5}$	2/20	3.3	7.2	6.3	4.5	2.6	3.5
$1.6\times 10^{-5}$	6/20	3.7	7.6	7.8	1.5	1.2	5.2
$2.5\times 10^{-5}$	17/20	5.3	6.7	9.7	7.0	3.8	3.9
$6.3\times 10^{-5}$	20/20	7.6	12.8	11.8	9.9	5.3	8.6

Table 2: NFP test response across ground-truth temporal fractions

\beta^{2}

. “Flagged” counts the random directions passing the Bonferroni threshold (

|t|>4.0

) at each level. The last five columns give the response of directions with all temporal content in a single concept.

Subspace (dim)	Flagged (8)	Unflagged (6136)	Random baseline
temporal $W_{\tau}$ (5)	0.582	0.003	0.007
static $W_{\sigma}$ (100)	0.105	0.236	0.130
position $W_{p}$ (50)	0.051	0.106	0.065

Table 3: Mean fraction of SAE feature norm in each planted subspace, for flagged and unflagged features. The random baseline is

d_{\text{sub}}/768

Model	Feature Count	Baseline	Identified	Intermediate	Low Static	Random
VideoMAE finetuned	109	0.629	0.572	0.595	0.602	0.602
V-JEPA2	62	0.681	0.578	0.558	0.540	0.562
TimeSformer	16	0.495	0.460	0.437	0.455	0.490

Table 4: Columns show all-class accuracy. Baseline: no features erased. Identified: NFP-identified temporal features erased. Low Static: Drawn from features whose covariance with every temporal concept gives

|t|<2

(compared to the temporal threshold of

|t|>4+

), span erased. Intermediate: Valid features (see Appendix H ) that are not identified or low static. Random: Randomly selected from all non-flagged features including non-valid features, performance averaged across 3 seeds. Random can also be seen as an imperfect proxy for features active on SSv2 but not on the stimulus set. TimeSformer SAE features are drawn from layer 10, not layer 11, because the layer 11 representation is not used for SSv2 classification except for the CLS token.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Arch.	Training	Frames	Res.	$\ell_{p}$	Grid	$d$	Layers
VideoMAE	ViT-B	MAE pretrain	16	$224^{2}$	2	$8\!\times\!14\!\times\!14$	768	2, 5, 11
VideoMAE-FT	ViT-B	+ SSv2 finetuning	16	$224^{2}$	2	$8\!\times\!14\!\times\!14$	768	2, 5, 11
V-JEPA2	ViT-L	self-sup. JE	16	$256^{2}$	2	$8\!\times\!16\!\times\!16$	1024	5, 11, 23
TimeSformer	ViT-B	+ SSv2 finetuning	8	$224^{2}$	1	$8\!\times\!14\!\times\!14$	768	2, 5, 11
DINOv2 †	ViT-B	image self-sup.	16	$224^{2}$	—	$16\!\times\!16$	768	final

Appendix

Table 5: Model specifications. Grid : spatiotemporal token grid (

T/\ell_{p}\times H_{p}\times W_{p}

d

: representation dimension per token. Layers : transformer layers from which post-MLP residual-stream activations are extracted. † DINOv2 processes each of the 16 frames independently with no cross-frame attention and serves as a negative control. TimeSformer’s 8 frames are every second frame of the 16-frame clip, and NFP covariance for TimeSformer is computed over these 8 time steps. Model identifiers are listed in Table 6 .

Model	Identifier
VideoMAE	MCG-NJU/videomae-base-ssv2
VideoMAE-FT	MCG-NJU/videomae-base-finetuned-ssv2
V-JEPA2	facebook/vjepa2-vitl-fpc16-256-ssv2
TimeSformer	facebook/timesformer-base-finetuned-ssv2
DINOv2	facebook/dinov2-base

Appendix

Table 6: HuggingFace checkpoint identifiers for each model.

Hyperparameter	Value
Expansion factor	$8\times$
$\ell_{1}$ penalty $\lambda$	0.1
Dead-feature coefficient	0.03
AuxK $\alpha$	0.0
Learning rate	$2\times 10^{-4}$
Batch size	4096

Appendix

Table 7: Hyperparameters of the standard

\ell_{1}

SAEs, identical for every model: finetuned and pretrained VideoMAE, TimeSformer at layers 11 and 10, V-JEPA2, DINOv2, and the synthetic control. Only the dictionary size, corpus, and step count vary (Table 8 ); the synthetic SAE uses 5,000 steps because its corpus is smaller.

SAE	Layer	$d\to D$	Tokens per clip	Training tokens
VideoMAE finetuned	11	$768\to 6144$	1568	${\sim}6.3$ M
VideoMAE	11	$768\to 6144$	1568	${\sim}6.3$ M
TimeSformer	11	$768\to 6144$	1569 (incl. CLS)	${\sim}6.3$ M
TimeSformer	10	$768\to 6144$	1569 (incl. CLS)	${\sim}6.3$ M
V-JEPA2	23	$1024\to 8192$	2048	${\sim}8.2$ M
DINOv2	final	$768\to 6144$	256 per frame	${\sim}16$ M

Appendix

Table 8: Per-SAE corpus and dictionary size. Every SAE is trained on the activations of 4000 SSv2 training clips from its own model, except the synthetic SAE, which uses an 80/20 train/val split of the 3000 generated videos. DINOv2 processes each frame independently as 256 spatial patch tokens.

SAE	FVE	$L_{0}$	Alive on SSv2
VideoMAE finetuned (L11)	0.59	40.5	6067 / 6144 (98.7%)
TimeSformer L11	0.90	12.8	5536 / 6144 (90.1%)
TimeSformer L10	0.94	3.4	5577 / 6144 (90.8%)
V-JEPA2 (L23)	0.67	57.5	8183 / 8192 (99.9%)
DINOv2 (final)	$\sim$ 0.80	$\sim$ 28	6122 / 6144 (99.6%)
Synthetic control	$\sim$ 0.51	$\sim$ 71	n/a

Appendix

Table 9: FVE (fraction of variance explained)

=1-\text{FVU}

, on held-out activations. FVE and

L_{0}

come from training logs, except for the DINOv2 and synthetic SAEs, which were not logged and are recomputed from their checkpoints. The recomputation gives FVE about 0.04 above the logged metric (0.63 vs. 0.59 on VideoMAE finetuned), so those two FVE values are marked approximate. Alive on SSv2 counts features that fire at least once on SSv2 validation. The synthetic SAE has no SSv2 count because it is trained on synthetic control activations.

Model	Axes (valid / $D$ )	SAE (valid / $D$ )
VideoMAE finetuned	768 / 768	637 / 6144
VideoMAE	768 / 768	571 / 6144
TimeSformer L11	768 / 768	93 / 6144
V-JEPA2	1024 / 1024	2783 / 8192
DINOv2	768 / 768	597 / 6144

Appendix

Table 10: Valid features (finite

t

in the one-sample t-test) for the axes and SAE dictionaries of each model. Every axes dimension is valid. The sparse SAE dictionaries have most features non-valid on ball stimulus, which is expected since the SAEs were trained on SSv2 and the synthetic ball stimulus is both out of distribution and involves extremely simple videos. SAE counts are on the decorrelated stimulus.

Model	$D$	Layer	speed	$v_{x}$	$v_{y}$	$\|a\|$	direction	any	%
VideoMAE finetuned	768	2	1	2	21	499	15	507	66.0
		5	258	244	144	522	52	672	87.5
		11 (final)	352	359	151	496	89	707	92.1
VideoMAE	768	2	3	3	27	506	27	521	67.8
		5	8	24	33	559	32	576	75.0
		11 (final)	6	23	58	515	29	540	70.3

Appendix

Table 11: Representation space dimension axes flagged by the NFP test at a given layer at the Bonferroni threshold

p<0.05/D

D

means the number of directions

w

tested, which here is equivalent to the number of dimensions. Per-profile columns count axes significant for that

\tau_{k}

, and an axis can be significant for several. “any” counts dimensions significant for at least one profile.

			$\max_{k}\|t\|$ , single-concept directions
$\beta^{2}$	Flagged	Median $\max_{k}\|t\|$	speed	$v_{x}$	$v_{y}$	$\|a\|$	direction
0 (random)	0/100	1.5	–	–	–	–	–
0 (position only)	0/50	1.5	–	–	–	–	–
0 (constant only)	0/50	1.4	–	–	–	–	–
$10^{-5}$	2/20	3.3	7.2	6.3	4.5	2.6	3.5
$1.6\times 10^{-5}$	6/20	3.7	7.6	7.8	1.5	1.2	5.2

Appendix

Table 12: Full projection-fraction sweep across all 26 log-spaced

\beta^{2}

levels plus the three

\beta^{2}=0

null sets. This is the complete version of Table 2 , which lists a representative subset. “Flagged” counts the random directions passing the Bonferroni threshold, and the last five columns give the

\max_{k}|t|

of the single-concept directions.

			$\|\cos\|$ with planted concept direction
Feature	Flagged for	ProjFrac $W_{\tau}$	speed	$v_{x}$	$v_{y}$	$\|a\|$	direction
feat00601	speed, $v_{x}$ , $v_{y}$	0.714	0.026	0.292	0.763	0.203	0.069
feat03882	speed, $v_{x}$ , $v_{y}$	0.659	0.080	0.290	0.731	0.138	0.120
feat00627	speed, $v_{x}$ , $v_{y}$	0.648	0.028	0.477	0.620	0.157	0.103
feat03197	all five	0.608	0.121	0.610	0.174	0.027	0.437
feat01793	$v_{x}$ , $v_{y}$ , direction	0.602	0.102	0.581	0.156	0.066	0.474

Appendix

Table 13: The 8 flagged synthetic-SAE features. Every flagged feature places

0.41

–

0.71

of its norm in the planted 5-d temporal subspace. The absolute cosine with each planted concept direction is listed, with the dominant direction in bold.

Geometry	Flagged	PF (flagged)	PF (unflagged)	MaxCos range
orthonormal ( $c=0$ )	8	0.582	0.003	0.57–0.76
pairwise $c=0.25$	17	0.351	0.004	0.33–0.59
pairwise $c=0.5$	20	0.237	0.005	0.28–0.55

Appendix

Table 14: NFP results under correlated temporal directions. PF is the mean ProjFrac of flagged/unflagged features, computed against an orthonormal basis of the temporal subspace’s span. The last column counts flagged features with

\mathrm{ProjFrac}<0.05

(false positives).

Model	Feature Count	Baseline	Identified	Low Static	Random
VideoMAE finetuned	109	0.629	0.634	0.631	0.627
V-JEPA2	62	0.681	0.678	0.680	0.675
TimeSformer L10	16	0.495	0.494	0.489	0.495

Appendix

Table 15: Performance at amplification

\alpha=3

. Columns show all-class accuracy. TimeSformer SAE features are drawn from layer 10, not layer 11.

Model	Feature Count	Baseline	$\Delta_{\mathrm{temporal}}$	$\Delta_{\mathrm{rnd}}$	Excess $=\Delta_{\mathrm{temporal}}-\Delta_{\mathrm{rnd}}$
VideoMAE finetuned	109	0.954	$-0.021$	$-0.009$	$-0.012$
V-JEPA2	62	0.951	$-0.017$	$-0.022$	$+0.005$
TimeSformer L10	16	0.914	$-0.011$	$-0.001$	$-0.010$

Appendix

Table 16: Binary classification performance under span erasure.

Model	Feature	Tag	Pair	Flip rate (top-1)	Rank-based rate
VideoMAE finetuned	feat00250	$v_{x}$	spin continue/stop	0.67	1.00
	feat00214	$v_{x}$	camera left/right	0.62	0.71
	feat02818	speed	fall fast/slow	0.50	0.62
	feat05077	$v_{x}$	fall fast/slow	0.50	0.67
V-JEPA2	feat04854	$\|a\|$	spin continue/stop	0.67	0.83
	feat03101	$\|a\|$	spin continue/stop	0.58	0.92

Appendix

Table 17: All (feature, pair) entries with top-1 flip rate

\geq 0.5

(flippers), per model, for the hand-picked opposing pairs.

Model	Feature	Tag	Pair	Flip rate (top-1)	Rank-based rate
VideoMAE finetuned	feat05622	speed	tilt fall/stay	0.67	1.00
VideoMAE finetuned	feat02951	$v_{x}$	poke move/still	0.54	0.83
V-JEPA2	feat03759	$\|a\|$	roll down/up	0.54	0.75
V-JEPA2	feat04022	$\|a\|$	poke move/still	0.50	0.83
TimeSformer L10	-	-	-	-	-

Appendix

Table 18: All (feature, pair) entries with top-1 flip rate

\geq 0.5

, per model, on the randomly drawn opposing pairs. TimeSformer’s identified features produce no flipper (maximum

0.46

	Hand-picked pairs		Random pairs
Model	Identified	Low static	Identified	Low static
VideoMAE finetuned	4	0	2	0
V-JEPA2	3	3	2	1
TimeSformer	3	1	0	3

Appendix

Table 19: Count of flippers (feature, pair entries with top-1 flip rate

\geq 0.5

) that are temporal and low static, on the hand-picked pairs and the randomly drawn pairs. Note that no flippers for both the hand-picked pairs and random pairs, temporal nor static, are for negative control pairs. This may suggest that features encoding negative control pair related concepts do not significantly encode concepts related to the other pairs tested.

Key	Axis	Positive class	Negative class
push_lr	$v_{x}$	Pushing [sth] from left to right	Pushing [sth] from right to left
pull_lr	$v_{x}$	Pulling [sth] from left to right	Pulling [sth] from right to left
cam_lr	$v_{x}$	Turning the camera left	Turning the camera right
move_ud	$v_{y}$	Moving [sth] up	Moving [sth] down
cam_ud	$v_{y}$	Turning the camera upwards	Turning the camera downwards
fall_speed	speed	[Sth] falling like a rock	[Sth] falling like a feather or paper

Appendix

Table 20: The 10 screen pairs. “Turning the camera …” classes abbreviate “…while filming [something]”. Depth pairs serve as internal negative controls to check whether temporal features flip non-motion concepts, which is possible due to feature polysemanticity.

Key	Axis	Positive class	Negative class
push_falloff	$\|a\|$	Pushing [sth] so it almost falls off	Pushing [sth] so it falls off the table
poke_move	speed	Poking [sth] so it slightly moves	Poking [sth] so it almost doesn’t move
roll_updown	$v_{x}$	Letting [sth] roll down a slanted surface	Letting [sth] roll up, so it rolls back
throw_catch	$\|a\|$	Throwing [sth] in the air and catching it	Throwing [sth] in the air, letting it fall
move_apart	$v_{x}$	Moving [sth] and [sth] closer	Moving [sth] and [sth] away
lift_slide	$\|a\|$	Lifting a surface, not enough to slide	Lifting a surface until [sth] slides down

Appendix

Table 21: The 10 randomly drawn screen pairs (draw seed 0, 91 draws). Class texts are abbreviated. Control pairs serve a similar purpose as the depth pairs in Table 20 , as an internal control to check whether temporal features flip pairs that are non-motion related, which is possible due to feature polysemanticity. Note that the random draw coincidentally selected ‘cam_appr’, which also appears as a hand-picked depth control. This is a coincidence of the draw, not a deliberate repeat.

Pair	Mean within-video covariance	Mean within-video correlation
speed– $\|a\|$ (target)	$+0.146$	$+0.876$
$\|a\|$ –direction	$-0.000$	$+0.124$
speed–direction	$-0.000$	$+0.117$
$v_{x}$ –direction	$-0.000$	$+0.066$
$v_{y}$ –direction	$+0.001$	$+0.060$
speed– $v_{x}$	$+0.001$	$+0.014$

Appendix

Table 22: Within-video couplings of the strongly entangled NFP set, recomputed from the 3000 rendered clips. The speed–

|a|

pair has a correlation of

0.88

as rendered (target 0.9). Every other pair has covariance

\leq 0.003

. Acceleration magnitude has a smaller within-video scale than speed (std

0.20

versus

0.65

), since

|a(t)|=|v(t{+}1)-v(t)|

is a difference of adjacent velocities.

Model	Layer	speed	$v_{x}$	$v_{y}$	$\|a\|$	$\|a\|$ also speed
VideoMAE finetuned	11	51	17	20	59	45 (76%)
VideoMAE	11	32	0	0	43	26 (60%)
TimeSformer	11	2	0	0	2	2 (100%)
V-JEPA2	23	64	0	0	62	50 (81%)
DINOv2	final	0	0	0	0	0

Appendix

Table 23: NFP features flagged on the strongly entangled set. Every test uses the model’s

\ell_{1}

SAE dictionary applied at the residual stream after the listed layer’s MLP; DINOv2 uses its SAE on the final-layer spatial patch tokens. Each feature is called significant by the within-video covariance

t

-test at Bonferroni

p<0.05/D

with non-zero covariance, where

D

is the full dictionary size (6144, or 8192 for V-JEPA2).

Figure 1: Frame decomposition of four example NFP test videos across 4 time stamps. The NFP test videos vary in ball movement pattern and starting position but not colors or sizes of the ball or the background.

Spatiotemporal Transformers

See all 23 papers

What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models
Sep 1, 2026Sharon S. Musa, Fereshteh Forghani, Harrish Thasarathan +3Video Representation Learning Vision Foundation Models
Lassonde School of Engineering, York University, Toronto, Canada · Vector Institute, Toronto, Canada · FAIR, Meta Superintelligence Labs +2
Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs
Oct 1, 2026Youngwoo Shin, Yusung Ro, Minseo Kim +1Video-Language Models Activation Steering
Korea Advanced Institute of Science and Technology (KAIST)
Interpreting Video Representations with Spatio-Temporal Sparse Autoencoders
Apr 5, 2026Atahan Dokme, Sriram VishwanathSparse Autoencoders Video Representation Learning
Georgia Institute of Technology, Atlanta, Georgia, USA

Sep 1, 2026J/K move · Enter open · S save

What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models

Sharon S. Musa, Fereshteh Forghani, Harrish Thasarathan, Sonia Joseph+2

Lassonde School of Engineering, York University, Toronto, Canada · Vector Institute, Toronto, Canada · FAIR, Meta Superintelligence Labs+2

Video Representation Learning Vision Foundation Models Physical Reasoning in Video Camera Motion Understanding Representation Probing Video Foundation Models+1

Open paper AskPDF arXiv

Self-supervised video foundation models learn rich spatiotemporal representations, yet it remains unclear what visual concepts these representations encode, where they emerge across transformer layers, and how they are geometrically organized. In this work, we tackle these three questions through a systematic layer-wise analysis of V-JEPA 2 and VideoMAE-v2. We leverage lightweight probes trained to discover three temporally grounded properties: (i) camera motion understanding, (ii) intuitive physics, and (iii) anomaly detection. Both models encode camera motion, with best results (

>90

ROC AUC) emerging at 60-70% of network depth, and achieve moderate anomaly detection performance (

>60

ROC AUC), but remain near chance on intuitive-physics tasks, suggesting a limited encoding of deeper physical reasoning. Beyond classification, we find that temporal features from individual videos form smooth low-dimensional trajectories in representation space, suggesting that camera motion is not only linearly decodable but also geometrically organized. Based on these results, we apply geometry-aware spline-based steering in the model's latent representations to interpolate camera motion, yielding steered videos with smoother trajectories and more coherent temporal progression than linear interpolation.

Abstract

Spatiotemporal Transformers

Explore similar work

What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Models

Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMs

Interpreting Video Representations with Spatio-Temporal Sparse Autoencoders