We present the first systematic study of Sparse Autoencoders (SAEs) on video representations. Standard SAEs decompose video into interpretable, monosemantic features but destroy temporal coherence: hard TopK selection produces unstable feature assignments across frames, reducing autocorrelation by 36%. We propose spatio-temporal contrastive objectives and Matryoshka hierarchical grouping that recover and even exceed raw temporal coherence. The contrastive loss weight controls a tunable trade-off between reconstruction and temporal coherence. A systematic ablation on two backbones and two datasets shows that different configurations excel at different goals: reconstruction fidelity, temporal coherence, action discrimination, or interpretability. Contrastive SAE features improve action classification by +3.9 pp over raw features and text-video retrieval by up to 2.8 x R@1. A cross-backbone analysis reveals that standard monosemanticity metrics contain a backbone-alignment artifact: both DINOv2 and VideoMAE produce equally monosemantic features under an independent (CLIP) similarity space. Targeted feature ablation shows that contrastive training concentrates the probe's predictive signal into a small number of identifiable features. Supplementary material, code, configurations and evaluation scripts are available at https://github.com/atahandokme/spatio-temporal-sparse-autoencoders-video.
Figures & tables
Figure 1. Contrastive pairing structures for the three spatio-temporal SAE variants. (a) Temporal : pairs the same spatial position across consecutive frames. (b) Separate : applies independent temporal (red, cross-frame) and spatial (blue, within-frame) contrastive losses. (c) Raster : serializes patches row-by-row, then across frames; at each frame boundary (orange dashed) the loss couples the last patch of frame t to the first of frame t+1 — a cross-frame boundary pair, not a same-position temporal correspondence.
Backbone
Variant
MS (own)
MS (CLIP)
DINOv2
Standard
0.365
0.681
DINOv2
Temporal
0.327
0.671
DINOv2
Separate
0.328
0.671
DINOv2
Raster
0.366
0.683
DINOv2
Temporal+M
0.326
0.670
DINOv2
Separate+M
0.317
0.668
Table 1. MonoSemanticity Score under different similarity backbones. Across all variants, VideoMAE reaches own-MS 0.969 – 0.974 against DINOv2’s 0.317 – 0.366 , a gap of roughly 2.7× ; under an independent CLIP-based similarity space every entry collapses to 0.67 – 0.68 , revealing a metric alignment artifact. Own-MS is comparable within a backbone but not across backbones.
Figure 2. SAE features on DINOv2/SSv2. (a) A scene feature activating on flat surfaces across diverse actions (holding, pushing, pouring, dropping). (b) An action-correlated feature firing on “Pushing [something] off the table,” with spatial activation concentrated on the pushing hand. (c, d) Temporal comparison: standard SAE features flicker across frames, while spatio-temporal SAE variants maintain more consistent activations.
Variant
R2↑
Lag-1 ↑
MS ↑
Probe ↑
Raw DINOv2
—
0.435
—
17.4%
Standard SAE
0.454
0.277
0.359
19.1%
Post-hoc smoothing of the standard SAE
EMA ( α=0.3 )
—
0.280
—
7.1%
EMA ( α=0.5 )
—
0.210
—
7.8%
Temporal Union
—
0.150
—
8.2%
Table 2. Balanced SAE variants on DINOv2/SSv2 (10K clips), selected from 29 configurations over λ∈{0.01,0.05,0.5} and τ∈{0.05,0.2,0.5} ; full results are in the Supplementary Material. Higher contrastive weight can further improve probe accuracy at the cost of R2 ; Separate at λ=0.5 reaches 21.3% (Table 3 ). All variants outperform raw features with zero dead features. Probe uses an 80/20 split. † Coherence-optimized: lag-1 exceeds raw DINOv2.
N
Standard
Temporal
Separate
Raster
Temp.+M
Sep.+M
Top
Rand
Top
Rand
Top
Rand
Top
Rand
Top
Rand
Top
Rand
0
19.1
20.2
21.3
19.3
18.8
19.7
10
18.1
19.2
10.9
20.4
19.6
21.1
17.8
19.3
16.4
18.9
12.9
19.7
50
16.3
19.2
13.4
20.1
16.9
21.1
13.3
19.3
13.0
18.8
11.0
19.5
100
17.4
19.2
16.1
19.9
15.9
20.9
12.8
18.5
12.9
18.6
12.4
19.1
500
17.3
18.7
14.1
17.4
14.4
19.4
8.5
14.9
10.9
17.1
10.3
16.4
Table 3. Targeted feature ablation on DINOv2/SSv2 (10K, no probe retraining, following Stevens et al. ( Stevens et al., 2025 ) ). We train a probe once, then zero the top- N features by weight magnitude and evaluate with the same weights. Baseline accuracies ( N=0 ) differ from Table 2 because each variant uses the checkpoint with the strongest ablation sensitivity (Separate: λ=0.5 , Raster: τ=0.5 , Temp.+M / Sep.+M: λ=0.01 ), not the best-probe configuration.
Model
Representation
R@1
R@5
DINOv2
Raw (768 → 512)
4.6%
15.2%
Reconstructed (768 → 512)
5.4%
16.6%
Standard SAE (6144 → 512)
13.0%
27.8%
Temporal Contr. (6144 → 512)
13.1 %
28.3 %
VideoMAE
Raw (768 → 512)
13.4%
26.6%
Reconstructed (768 → 512)
15.2%
27.6%
Table 4. Text-video retrieval on SSv2 (10K clips, 80/20 split, Ridge α=1.0 ). SAE-reconstructed (768d) controls for dimensionality. VideoMAE SAE features achieve the highest absolute R@1.
Figure 3. Qualitative text-video retrieval comparison (DINOv2/SSv2, Standard SAE). For each query, we show the top-5 retrieved clips using raw features (left) vs. SAE features (right). Green borders indicate exact template match (conservative: visually similar clips with different templates count as incorrect). SAE features retrieve 11/20 correct matches vs. 4/20 for raw features.
Goal
Variant
Achieved
Reconstruction fidelity
Raster, λ=0.01
R2=0.418
Temporal coherence
Temporal+M, τ=0.2
lag-1 =0.462
Action discrimination
Separate, λ=0.5
probe =21.3%
Interpretability (MS)
Raster, λ=0.5
MS =0.433
Table 5. Which variant to use for which goal (DINOv2/SSv2, 10K clips). No variant dominates every metric; the contrastive weight selects the operating point.