I Have a Stream: Making Self-Supervised Learning Work on Continuous Video

Ivan Martinović; Lukas Knobel; Yuki M. Asano

cs.CVSep 30, 2026

I Have a Stream: Making Self-Supervised Learning Work on Continuous Video

Authors: Ivan Martinović, Lukas Knobel, Yuki M. Asano

Organizations: Faculty of Electrical Engineering and Computing, University of Zagreb · Fundamental AI Lab, University of Technology Nuremberg

Abstract

Self-supervised learning draws inspiration from infant visual development, yet standard training pipelines bear little resemblance to it: images are independently sampled and globally shuffled across epochs. We study self-supervised learning from continuous video streams, where frames are consumed in temporal order using strict sliding-window batches, without global reshuffling or multi-epoch replay. To this end, we construct WT++, a 95-hour urban walking-tour video dataset for streaming pretraining. Combined with a comprehensive evaluation suite we find that contrastive and distillation-based methods struggle in this setting, while MAE is more robust but still falls short of standard i.i.d. pretraining. We find that high inter-batch similarity, caused by sliding-window consumption across consecutive batches, does not explain this gap. The main challenge is high intra-batch similarity, where frames within each batch are near-duplicates. To mitigate this, we propose StreamMAE, which preserves the core MAE reconstruction objective while adapting the input pipeline with stream-aware regularization and motion-biased crop selection. StreamMAE outperforms streaming baselines, matches i.i.d. MAE trained on the same video data, remains competitive with ImageNet-pretrained MAE, and scales positively as the pretraining stream grows from 12 to 95 hours.

Figures & tables

Figure 1: Left: StreamMAE adapts MAE to continuous video streams with stream-aware regularization and crop selection. Right: StreamMAE (ViT-S) improves dense performance over streaming SSL baselines (top) and improves with longer pretraining streams and a larger backbone (bottom).

Figure 2: Streaming pretraining setup. Frames are observed in temporal order, and each batch is formed as a sliding window over the stream. Here: $B=4$ and stride $s=2$ , but typically $s\ll B$ .

Figure 3: Pairwise DINOv2-L [ 37 ] feature similarities for 512 frames from WT++London show that consecutive frames at 15 FPS can be highly similar (left) or contain structured scene changes (block diagonal, middle), whereas randomly sampled frames are markedly more diverse (right).

	i.i.d. IN-1K	Streaming
Metric	i.i.d. IN-1K	IN-1K	WT++12h
$\mu_{\mathrm{intra}}$	0.004	0.004	0.665
$\mu_{\mathrm{inter}}$	0.325	0.989	1.000

Table 1: Batch similarity statistics.

Backbone	Training regime	IN-1K Acc@1 $\uparrow$	CS mIoU $\uparrow$	ADE20K mIoU $\uparrow$
ViT-S	Standard i.i.d.	77.4	64.0	26.9
ViT-S	Pre-shuffled stream	77.4	63.6	27.2
ViT-B	Standard i.i.d.	81.5	72.3	35.9
ViT-B	Pre-shuffled stream	81.6	73.6	36.5

Table 2: Fixed-order sliding-window training on diverse data. We pre-shuffle ImageNet-1K once and consume it as a fixed stream with stride

s=8

. The pre-shuffled stream matches standard i.i.d. MAE, suggesting that inter-batch similarity alone does not explain the streaming gap.

Figure 4: (a) Running average of consecutive batches gradient similarity, measured on last-block MLP parameters every 10 steps. (b) Dense downstream performance versus distance to standard i.i.d. MAE in mean gradient similarity. Methods closer to the i.i.d. MAE reference perform better.

Figure 5: Motion-biased crop selection computes patch-level differences between consecutive frames $\mathbf{x}_{t}$ and $\mathbf{x}_{t-1}$ , samples $K$ crop candidates $\mathcal{C}$ , and selects the crop $c_{t}^{\star}$ with the largest average $\ell_{1}$ score.

Pretraining Method	Pretraining Data	IN-1K [ 15 ] Acc @ 1 $\uparrow$	City [ 12 ] mIoU $\uparrow$	ADE [ 57 ] mIoU $\uparrow$	NYUv2 [ 46 ] RMSE $\downarrow$	KITTI [ 19 ] RMSE $\downarrow$
Random Init	–	71.9	47.7	16.7	${0.877}_{\pm 0.013}$	${6.108}_{\pm 0.028}$
Standard I.I.D. Setup
MAE – standard IID	WT++12h	77.0	63.5	25.9	${0.701}_{\pm 0.004}$	${4.070}_{\pm 0.006}$
MAE – standard IID	ImageNet-1K	77.4	64.0	26.9	${0.656}_{\pm 0.004}$	${3.854}_{\pm 0.008}$
Streaming Setup
MOCO-v3 [ 10 ]	WT++12h	68.7	52.0	19.4	${0.760}_{\pm 0.000}$	${4.882}_{\pm 0.054}$

Table 3: Benchmarking SSL methods under streaming pretraining (ViT-S). We compare WT++12h streaming methods to random initialization, i.i.d. MAE on WT++12h , and an iteration-matched ImageNet-1K i.i.d. MAE reference. † marks reproduced streaming-tailored methods.

Figure 6: Cumulative ablation study with ViT-S/16 pretrained on WT++12h . Each row adds one component to the previous setting. DataDrop gives modest gains, while stronger regularization and crop selection account for most of the dense-transfer improvement. CS denotes crop selection.

Pretraining Method	Encoder	Pretraining Data	IN-1K Acc @ 1 $\uparrow$	Cityscapes mIoU $\uparrow$	ADE20K mIoU $\uparrow$	NYUv2 RMSE $\downarrow$	KITTI RMSE $\downarrow$
Random Init	ViT-B	–	78.5	51.0	17.5	${0.881}_{\pm 0.002}$	${5.818}_{\pm 0.137}$
Standard I.I.D. Setup
MAE – standard IID	ViT-B	WT++12h	81.2	67.8	32.9	${0.675}_{\pm 0.004}$	${3.855}_{\pm 0.048}$
MAE – standard IID	ViT-B	ImageNet-1K	81.5	72.3	35.9	${0.588}_{\pm 0.003}$	${3.771}_{\pm 0.039}$
Streaming Setup
StreamMAE (ours)	ViT-S	WT++12h	77.5	63.8	26.1	${{0.694}}_{\pm{0.003}}$	${{3.976}}_{\pm{0.012}}$

Table 4: Increasing encoder size . StreamMAE (ViT-B) improves over both StreamMAE (ViT-S) and the MAE baseline, while closing most of the gap to standard i.i.d. MAE. † marks our reproduction.

Table 11

Method	WT duration	IN-1K Acc@1 $\uparrow$	CS (mIoU) $\uparrow$	ADE (mIoU) $\uparrow$
StreamMAE	WT++95h	82.0	74.0	36.5
Avg. {40,60,80,100}%	WT++95h	81.9	75.5	37.0
Avg. {60,100}%	WT++95h	81.9	75.3	37.3

Table 7: Checkpoint averaging on the same ViT-B WT++95h trajectory improves dense transfer with negligible change in IN-1K accuracy.

Table 13

Method	ViT-S/16		ViT-B/16
Method	Acc@1 $\uparrow$	Acc@5 $\uparrow$	Acc@1 $\uparrow$	Acc@5 $\uparrow$
MAE – standard i.i.d.	40.0	63.0	46.4	69.0
MAE (streaming)	33.4	55.8	37.3	59.6
StreamMAE	40.2	63.6	47.1	69.9

Table 10: ImageNet-1K attentive probing evaluation after pretraining on WT++12h .

Pretraining Data	Method	CS (mIoU) $\uparrow$	ADE (mIoU) $\uparrow$	NYUv2 (RMSE) $\downarrow$	KITTI (RMSE) $\downarrow$
HD-EPIC [ 38 ]	MAE – standard i.i.d.	65.2	30.7	${0.681}_{\pm 0.001}$	${4.184}_{\pm 0.081}$
	MAE (streaming)	63.2	29.3	${0.711}_{\pm 0.000}$	${4.206}_{\pm 0.003}$
	StreamMAE (ours)	65.6	32.0	${\mathbf{0.652}}_{\pm\mathbf{0.001}}$	${\mathbf{4.030}}_{\pm\mathbf{0.049}}$
CROWD [ 3 ]	MAE – standard i.i.d.	68.1	32.3	${0.681}_{\pm 0.000}$	${3.830}_{\pm 0.026}$
	MAE (streaming)	65.7	29.1	${0.704}_{\pm 0.004}$	${4.287}_{\pm 0.147}$
	StreamMAE (ours)	71.5	32.6	${\mathbf{0.644}}_{\pm\mathbf{0.001}}$	${\mathbf{3.756}}_{\pm\mathbf{0.046}}$

Table 11: Generalization across pretraining domains using ViT-B/16 .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Training regime	$\mu_{\mathrm{intra}}$	$\mu_{\mathrm{inter}}$	ViT-S/16		ViT-B/16
Training regime	$\mu_{\mathrm{intra}}$	$\mu_{\mathrm{inter}}$	ADE20K (mIoU) $\uparrow$	Cityscapes (mIoU) $\uparrow$	ADE20K (mIoU) $\uparrow$	Cityscapes (mIoU) $\uparrow$
i.i.d.	0.171	0.738	25.9	63.5	32.9	67.8
Pre-shuffled stream	0.171	0.999	26.0	63.9	32.3	68.8
Chronological stream	0.665	1.000	23.9	61.3	26.2	58.2

Appendix

Table 12: Intra-/inter-batch similarity and downstream performance for ViT-S/16 and ViT-B/16 pretrained on WT++12h .

Method	ViT-S/16			ViT-B/16
Method	IN-1K Acc@1 $\uparrow$	ADE20K mIoU $\uparrow$	Cityscapes mIoU $\uparrow$	IN-1K Acc@1 $\uparrow$	ADE20K mIoU $\uparrow$	Cityscapes mIoU $\uparrow$
MAE – standard i.i.d.	${77.02}_{\pm 0.02}$	${{25.9}}_{\pm{0.1}}$	${{63.4}}_{\pm{0.7}}$	${{81.12}}_{\pm{0.10}}$	${32.8}_{\pm 0.1}$	${68.1}_{\pm 0.4}$
MAE (streaming)	${77.02}_{\pm 0.09}$	${24.1}_{\pm 0.2}$	${60.6}_{\pm 0.6}$	${80.05}_{\pm 0.04}$	${26.6}_{\pm 0.2}$	${58.9}_{\pm 0.5}$
StreamMAE	${{77.56}}_{\pm{0.05}}$	${25.8}_{\pm 0.3}$	${63.3}_{\pm 0.6}$	${{81.12}}_{\pm{0.00}}$	${{32.9}}_{\pm{0.6}}$	${{69.5}}_{\pm{0.4}}$

Appendix

Table 13: Results across repeated downstream fine-tuning runs for fixed ViT-S/16 and ViT-B/16 checkpoints pretrained on WT++12h . We report mean and standard deviation over two seeds for ImageNet-1K and three seeds for ADE20K and Cityscapes.

Method	Batch size	DataDrop	IN-1K Acc@1 $\uparrow$	CS (mIoU) $\uparrow$	ADE (mIoU) $\uparrow$
StreamMAE (ViT-B)	$B=512$	✗	81.7	73.6	36.4
	$B=256$	✗	81.8	73.3	35.4
	$B=128$	✗	81.4	72.2	34.2

Appendix

Table 14: Effect of reducing batch duration without DataDrop . With ViT-B/16 pretrained on WT++50h at 3.75 FPS, smaller batches correspond to shorter and more temporally correlated video windows, yet performance degrades only gradually.

Method	Batch size	DataDrop (75%)	IN-1K Acc@1 $\uparrow$	CS (mIoU) $\uparrow$	ADE (mIoU) $\uparrow$
MOCOv3 [ 10 ] baseline (ViT-S)	$B=2048$	✗	66.3	51.7	18.6
MOCOv3 [ 10 ] baseline (ViT-S)	$B=2048$	✓	68.7	52.0	19.4
MAE [ 26 ] baseline w/ Orthogonal-AdamW [ 22 ] (ViT-S)	$B=2048$	✗	75.2	55.3	20.6
	$B=512$	✗	75.5	58.6	21.5
	$B=2048$	✓	75.9	58.3	22.5
MAE [ 26 ] baseline (ViT-S)	$B=2048$	✗	76.6	57.9	23.2

Appendix

Table 15: Effect of DataDrop on streaming baselines. We form a local window of

B=2048

samples and backpropagate through a randomly selected

25\%

subset. This generally improves over using the full window and matches or improves the

B=512

setting.

Method	WT duration	IN-1K Acc@1 $\uparrow$	CS (mIoU) $\uparrow$	ADE (mIoU) $\uparrow$
StreamMAE (ViT-B)	WT++12h +Budapest	81.1	69.4	32.5
	WT++25h +Budapest	81.5	72.2	35.2
	WT++50h	81.9	73.8	35.6

Appendix

Table 16: Controlling for the final-video effect. Appending the same final video to shorter streams does not match the full WT++50h stream.

Method	Masking Strategy	IN-1K Acc@1 $\uparrow$	CS (mIoU) $\uparrow$	ADE (mIoU) $\uparrow$
StreamMAE (ViT-B)	Block Masking [ 55 ] (75%)	80.9	68.1	32.8
	I-JEPA [ 4 ] / Bootleg [ 33 ]	80.4	66.9	31.7
	Random Uniform [ 26 ] (75%)	81.1	69.0	32.7

Appendix

Table 17: Effect of masking strategy for StreamMAE with ViT-B/16. Standard random uniform masking remains the strongest overall choice.

Method	AdamW $\beta_{1}$	IN-1K Acc@1 $\uparrow$	CS (mIoU) $\uparrow$	ADE (mIoU) $\uparrow$
StreamMAE (ViT-B)	$0$	81.0	67.1	31.8
StreamMAE (ViT-B)	$0.9$	81.1	69.0	32.7

Appendix

Table 18: Effect of AdamW first-moment momentum for StreamMAE with ViT-B/16. Our standard setting

\beta_{1}=0.9

performs better than deactivating momentum.

Components				Downstream performance
DataDrop	Color Jit. & Drop Path	Two-stage Cropping	Motion-biased Crop Selection	Cityscapes $\uparrow$	ADE20K $\uparrow$	KITTI $\downarrow$
–	–	–	–	60.8	23.2	${4.307}_{\pm 0.046}$
✓	–	–	–	61.3	23.9	${4.211}_{\pm 0.137}$
✓	✓	–	–	62.6	25.1	${4.025}_{\pm 0.039}$
✓	✓	✓	–	62.8	26.1	${4.074}_{\pm 0.048}$
✓	–	✓	✓	62.4	24.7	${4.022}_{\pm 0.123}$

Appendix

Table 19: Additional ablation of StreamMAE components with ViT-S/16 pretrained on WT++12h . The extra rows remove either DataDrop or regularization from the full method to isolate their roles.

Method	ADE (mIoU) $\uparrow$	CS (mIoU) $\uparrow$	NYUv2 (RMSE) $\downarrow$	KITTI (RMSE) $\downarrow$
StreamMAE w/o motion-biased selection	33.9	71.5	${\mathbf{0.621}}_{\pm\mathbf{0.004}}$	${3.646}_{\pm 0.053}$
StreamMAE (ours)	34.5	72.2	${\mathbf{0.621}}_{\pm\mathbf{0.001}}$	${\mathbf{3.583}}_{\pm\mathbf{0.012}}$

Appendix

Table 20: Ablation of motion-biased crop selection with ViT-B/16 pretrained on KrishnaCAM. Both variants use the same frames, pretraining budget, and remaining StreamMAE configuration.

Method	Pretraining data	IN-1K Acc@1 $\uparrow$	CS (mIoU) $\uparrow$	ADE (mIoU) $\uparrow$
Standard i.i.d. MAE (ViT-S)	ImageNet-1K	78.5	65.9	28.5
StreamMAE (ViT-S)	WT++95h	78.4	68.0	29.5
Standard i.i.d. MAE (ViT-B)	ImageNet-1K	82.5	75.2	39.8
StreamMAE (ViT-B)	WT++95h	82.0	74.0	36.5
StreamMAE (ViT-B, avg. {60,100}%; see Table 7 )	WT++95h	81.9	75.3	37.3

Appendix

Table 21: Longer i.i.d. ImageNet reference. Unlike the main benchmarking tables, where ImageNet-1K i.i.d. MAE is iteration-matched to WT++12h , this table matches the training length of the WT++95h streaming runs, approximately

650\mathrm{k}

iterations. StreamMAE trained on WT++95h remains competitive with this longer i.i.d. ImageNet reference, particularly for ViT-S and Cityscapes transfer.

Figure 7: Running average of cosine similarity between gradients from consecutive batches, measured on last-block MLP parameters every 10 steps.

Figure 8: Downstream performance versus the distance from standard i.i.d. MAE in mean gradient cosine similarity between consecutive batches throughout training. Extending Figure 4 , we include MAE trained on pre-shuffled & streaming WT++12h ( Table 12 ), StreamMAE with Orthogonal-AdamW, and streaming MAE using only motion-biased crop selection. All models use a ViT-S/16 backbone and are pretrained on WT++12h . All streaming variants use

B=512

without DataDrop .

Stream	# Videos	Duration
WT++12h	1	12.0h	London stream
WT++25h	10	24.8h	WT++12h + 9 videos from original WalkingTours [ 49 ]
WT++50h	26	50.0h	WT++25h + 16 appended videos
WT++95h	58	94.5h	Full WT++ collection

Appendix

Table 22: Construction of the ordered WT++ streams used in scaling experiments. Each stream is a prefix of the next one.

Stream stage	# New videos	Cities / videos in training order
WT++12h	1	London
WT++25h	9	Venice, Amsterdam, Singapore, Istanbul, Bangkok, Stockholm, Kuala Lumpur, Zurich, Chiang Mai
WT++50h	16	Frankfurt, Phnom Penh, Sevilla, Hoi An, Prague, Chemnitz, Marmaris, Dubrovnik, Mostar, Gibraltar, Barcelona, Valencia, Rhodes, Helsinki, Copenhagen, Budapest
WT++95h	32	Vilnius, Danang, Klaipeda, Liege, Marbella, Monschau, Naples, Nessebar, Port de Soller, Sarajevo, Arcadia, Timisoara, Warsaw, Malacca, Florence, Cardiff, Georgetown, Luxembourg, Alicante, Ipoh, Belfast, Kaliningrad, Maastricht, Valletta, Vienna, Saigon, Cadiz, Kyiv, Oxford, Sihanoukville, Chongqing, Tokyo

Appendix

Table 23: Construction of the ordered WT++ streams. Each row lists the new videos introduced at that stage, in training order (all videos from earlier stages are retained).

Experiment	Backbone	GPUs	Wall-clock time
StreamMAE pretraining on WT++12h	ViT-S/16	$2\times$ H100	$\sim$ 7h
StreamMAE pretraining on WT++12h	ViT-B/16	$2\times$ H100	$\sim$ 8h
MoCo v3 pretraining on WT++12h	ViT-S/16	$2\times$ H100	$\sim$ 18h
DINO pretraining on WT++12h	ViT-S/16	$2\times$ H100	$\sim$ 25h
ImageNet-1K fine-tuning	ViT-S/16	$4\times$ H100	$\sim$ 4h
ImageNet-1K fine-tuning	ViT-B/16	$4\times$ H100	$\sim$ 5h

Appendix

Table 24: Approximate compute requirements. Pretraining times are reported for WT++12h and exclude downstream fine-tuning.

Explore similar work

Sep 27, 2026cs.CV

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

Comparisons in video self-supervised learning often evaluate complete training recipes rather than isolating the method itself: architecture, objective, data exposure, schedule, scale, and decoder capacity can all vary at once. This makes it hard to identify which choices yield motion-prioritized representations, whose gains concentrate on frame-to-frame change while retaining useful appearance. We address this with a matched

4 \times 6 = 24

architecture-objective study at roughly 170M ~ 190M encoder scale on

\sim

1.7M OpenVid and Moments-in-Time v2 clips for 8 epochs, and propose TT-VidT. TT-VidT combines a DINOv3-initialized ViT-B/16 per-frame spatial path with a compact Temporal Transfer Layer, trained by Diff Compression to reconstruct target frames from a first-frame appearance anchor and frame-specific motion tokens. The sweep shows that TT3D with Diff Compression, not either component alone, enters the strongest motion-sensitive regime, and decoder ablations favor a compact video-pretrained decoder. In final comparison, TT-VidT leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, improving over the strongest non-TT row by 54% ~ 121%, while using 48% fewer encoder FLOPs than DisMo and 55% fewer than VideoMAE or V-JEPA2. HMDB51, IARD, and EPIC-Kitchens bound the claim.

Shih-Ying Yeh, Daniel Z. Kaplan, Xuehai Wang +3

National Tsing Hua University · Comfy Org Research · Karolinska Institutet +2

Topic pending

Ask PDF

Sep 30, 2026cs.CV

Image Classifiers are Efficient Self-Supervised Video Representation Learners

We introduce VideoMSN, a Masked Siamese Network framework for efficient self-supervised spatio-temporal representation learning in videos. Instead of relying on heavy 3D architectures or reconstruction-based autoencoders for learning with unlabeled data, we repurpose standard image Vision Transformers by representing videos as super images which are grids composed of frames sampled from videos. From each super image, we construct two views: one with spatial patch masking and the other with temporal frame masking, ensuring no information leakage across frames. A shared Vision Transformer (ViT) encoder aligns their embeddings using a masked Siamese loss, capturing both motion and appearance cues without reconstruction. Our decoder-free formulation leverages an image foundation model towards efficient video representation learning. Starting from pretrained DINO-v3 and DeiT-v3 image encoders, VideoMSN achieves state-of-the-art performance on Kinetics-400, UCF101, and HMDB51 while requiring up to

32\times

fewer and

160\times

fewer video pretraining epochs compared to prior video self-supervised learning methods. Our proposed approach also shows strong performance in low-shot classification, confirming the transferability of the learned representations in a label-scarce scenario. Project Page: https://cvir.github.io/projects/videomsn.

Owais Iqbal, Sudipta Sarkar, Shyam Marjit +3

Indian Institute of Technology Kharagpur, India · Indian Institute of Science Bangalore, India · École de technologie supérieure Montreal, Canada

Vision Transformer Image Classification Classifier

Ask PDF

May 29, 2026cs.CV

Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streams

In this work we introduce a novel approach to domain incremental learning, adapting models over time to evolving, non-stationary data. In contrast to other works, we do not attempt to avoid catastrophic forgetting, but rather allow it and exploit it. Our model combines a main task head with a self-supervised masked autoencoder (MAE) head. We then learn domain-specific LoRA adapters during incremental training. Each adapter specializes to its domain, naturally inducing forgetting on other domains in both heads. At inference, we perform online test-time training on the self-supervised MAE head to identify which LoRAs best matches the current input, so the model can `remember' the domain again. Our scheme is especially well-suited to real-world streaming data, such as video, where consecutive samples are highly correlated and domain shifts are gradual. We demonstrate our method on domain-incremental action recognition and semantic segmentation tasks.

Jonathan Swinnen, Tinne Tuytelaars

ESAT, KU Leuven, Leuven, Belgium

Class-Incremental Learning Streaming Video Low-Rank Adaptation Adapters 3D Masked Autoencoders Autoencoder Architectures Open-Vocabulary Action Recognition

Ask PDF