cs.CVSep 30, 2026

I Have a Stream: Making Self-Supervised Learning Work on Continuous Video

Authors: Ivan Martinović, Lukas Knobel, Yuki M. Asano

Organizations: Faculty of Electrical Engineering and Computing, University of Zagreb · Fundamental AI Lab, University of Technology Nuremberg

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Self-supervised learning draws inspiration from infant visual development, yet standard training pipelines bear little resemblance to it: images are independently sampled and globally shuffled across epochs. We study self-supervised learning from continuous video streams, where frames are consumed in temporal order using strict sliding-window batches, without global reshuffling or multi-epoch replay. To this end, we construct WT++, a 95-hour urban walking-tour video dataset for streaming pretraining. Combined with a comprehensive evaluation suite we find that contrastive and distillation-based methods struggle in this setting, while MAE is more robust but still falls short of standard i.i.d. pretraining. We find that high inter-batch similarity, caused by sliding-window consumption across consecutive batches, does not explain this gap. The main challenge is high intra-batch similarity, where frames within each batch are near-duplicates. To mitigate this, we propose StreamMAE, which preserves the core MAE reconstruction objective while adapting the input pipeline with stream-aware regularization and motion-biased crop selection. StreamMAE outperforms streaming baselines, matches i.i.d. MAE trained on the same video data, remains competitive with ImageNet-pretrained MAE, and scales positively as the pretraining stream grows from 12 to 95 hours.

Figures & tables

Figure 1: Left: StreamMAE adapts MAE to continuous video streams with stream-aware regularization and crop selection. Right: StreamMAE (ViT-S) improves dense performance over streaming SSL baselines (top) and improves with longer pretraining streams and a larger backbone (bottom).
Figure 2: Streaming pretraining setup. Frames are observed in temporal order, and each batch is formed as a sliding window over the stream. Here: B=4B=4 and stride s=2s=2 , but typically s≪Bs\ll B .
Figure 3: Pairwise DINOv2-L [ 37 ] feature similarities for 512 frames from WT++London show that consecutive frames at 15 FPS can be highly similar (left) or contain structured scene changes (block diagonal, middle), whereas randomly sampled frames are markedly more diverse (right).
 i.i.d. IN-1KStreaming
MetricIN-1KWT++12h
μintra\mu_{\mathrm{intra}}0.0040.0040.665
μinter\mu_{\mathrm{inter}}0.3250.9891.000
Table 1: Batch similarity statistics.
BackboneTraining regimeIN-1K Acc@1 ↑\uparrowCS mIoU ↑\uparrowADE20K mIoU ↑\uparrow
ViT-SStandard i.i.d.77.464.026.9
Pre-shuffled stream77.463.627.2
ViT-BStandard i.i.d.81.572.335.9
Pre-shuffled stream81.673.636.5
Table 2: Fixed-order sliding-window training on diverse data. We pre-shuffle ImageNet-1K once and consume it as a fixed stream with stride s=8s=8 . The pre-shuffled stream matches standard i.i.d. MAE, suggesting that inter-batch similarity alone does not explain the streaming gap.
Figure 4: (a) Running average of consecutive batches gradient similarity, measured on last-block MLP parameters every 10 steps. (b) Dense downstream performance versus distance to standard i.i.d. MAE in mean gradient similarity. Methods closer to the i.i.d. MAE reference perform better.
Figure 5: Motion-biased crop selection computes patch-level differences between consecutive frames xt\mathbf{x}_{t} and xt−1\mathbf{x}_{t-1} , samples KK crop candidates C\mathcal{C} , and selects the crop ct⋆c_{t}^{\star} with the largest average ℓ1\ell_{1} score.
Pretraining MethodPretraining DataIN-1K [ 15 ] Acc @ 1 ↑\uparrowCity [ 12 ] mIoU ↑\uparrowADE [ 57 ] mIoU ↑\uparrowNYUv2 [ 46 ] RMSE ↓\downarrowKITTI [ 19 ] RMSE ↓\downarrow
Random Init–71.947.716.70.877±0.013{0.877}_{\pm 0.013}6.108±0.028{6.108}_{\pm 0.028}
Standard I.I.D. Setup
MAE – standard IIDWT++12h77.063.525.90.701±0.004{0.701}_{\pm 0.004}4.070±0.006{4.070}_{\pm 0.006}
MAE – standard IIDImageNet-1K77.464.026.90.656±0.004{0.656}_{\pm 0.004}3.854±0.008{3.854}_{\pm 0.008}
Streaming Setup
MOCO-v3 [ 10 ]WT++12h68.752.019.40.760±0.000{0.760}_{\pm 0.000}4.882±0.054{4.882}_{\pm 0.054}
Table 3: Benchmarking SSL methods under streaming pretraining (ViT-S). We compare WT++12h streaming methods to random initialization, i.i.d. MAE on WT++12h , and an iteration-matched ImageNet-1K i.i.d. MAE reference. † marks reproduced streaming-tailored methods.
Figure 6: Cumulative ablation study with ViT-S/16 pretrained on WT++12h . Each row adds one component to the previous setting. DataDrop gives modest gains, while stronger regularization and crop selection account for most of the dense-transfer improvement. CS denotes crop selection.
Pretraining MethodEncoderPretraining DataIN-1K Acc @ 1 ↑\uparrowCityscapes mIoU ↑\uparrowADE20K mIoU ↑\uparrowNYUv2 RMSE ↓\downarrowKITTI RMSE ↓\downarrow
Random InitViT-B–78.551.017.50.881±0.002{0.881}_{\pm 0.002}5.818±0.137{5.818}_{\pm 0.137}
Standard I.I.D. Setup
MAE – standard IIDViT-BWT++12h81.267.832.90.675±0.004{0.675}_{\pm 0.004}3.855±0.048{3.855}_{\pm 0.048}
MAE – standard IIDViT-BImageNet-1K81.572.335.90.588±0.003{0.588}_{\pm 0.003}3.771±0.039{3.771}_{\pm 0.039}
Streaming Setup
StreamMAE (ours)ViT-SWT++12h77.563.826.10.694±0.003{{0.694}}_{\pm{0.003}}3.976±0.012{{3.976}}_{\pm{0.012}}
Table 4: Increasing encoder size . StreamMAE (ViT-B) improves over both StreamMAE (ViT-S) and the MAE baseline, while closing most of the gap to standard i.i.d. MAE. † marks our reproduction.
Table 11
MethodWT durationIN-1K Acc@1 ↑\uparrowCS (mIoU) ↑\uparrowADE (mIoU) ↑\uparrow
StreamMAEWT++95h82.074.036.5
Avg. {40,60,80,100}%WT++95h81.975.537.0
Avg. {60,100}%WT++95h81.975.337.3
Table 7: Checkpoint averaging on the same ViT-B WT++95h trajectory improves dense transfer with negligible change in IN-1K accuracy.
Table 13
MethodViT-S/16ViT-B/16
Acc@1 ↑\uparrowAcc@5 ↑\uparrowAcc@1 ↑\uparrowAcc@5 ↑\uparrow
MAE – standard i.i.d.40.063.046.469.0
MAE (streaming)33.455.837.359.6
StreamMAE40.263.647.169.9
Table 10: ImageNet-1K attentive probing evaluation after pretraining on WT++12h .
Pretraining DataMethodCS (mIoU) ↑\uparrowADE (mIoU) ↑\uparrowNYUv2 (RMSE) ↓\downarrowKITTI (RMSE) ↓\downarrow
HD-EPIC [ 38 ]MAE – standard i.i.d.65.230.70.681±0.001{0.681}_{\pm 0.001}4.184±0.081{4.184}_{\pm 0.081}
MAE (streaming)63.229.30.711±0.000{0.711}_{\pm 0.000}4.206±0.003{4.206}_{\pm 0.003}
StreamMAE (ours)65.632.00.652±0.001{\mathbf{0.652}}_{\pm\mathbf{0.001}}4.030±0.049{\mathbf{4.030}}_{\pm\mathbf{0.049}}
CROWD [ 3 ]MAE – standard i.i.d.68.132.30.681±0.000{0.681}_{\pm 0.000}3.830±0.026{3.830}_{\pm 0.026}
MAE (streaming)65.729.10.704±0.004{0.704}_{\pm 0.004}4.287±0.147{4.287}_{\pm 0.147}
StreamMAE (ours)71.532.60.644±0.001{\mathbf{0.644}}_{\pm\mathbf{0.001}}3.756±0.046{\mathbf{3.756}}_{\pm\mathbf{0.046}}
Table 11: Generalization across pretraining domains using ViT-B/16 .
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Training regimeμintra\mu_{\mathrm{intra}}μinter\mu_{\mathrm{inter}}ViT-S/16ViT-B/16
ADE20K (mIoU) ↑\uparrowCityscapes (mIoU) ↑\uparrowADE20K (mIoU) ↑\uparrowCityscapes (mIoU) ↑\uparrow
i.i.d.0.1710.73825.963.532.967.8
Pre-shuffled stream0.1710.99926.063.932.368.8
Chronological stream0.6651.00023.961.326.258.2
Appendix
Table 12: Intra-/inter-batch similarity and downstream performance for ViT-S/16 and ViT-B/16 pretrained on WT++12h .
MethodViT-S/16ViT-B/16
IN-1K Acc@1 ↑\uparrowADE20K mIoU ↑\uparrowCityscapes mIoU ↑\uparrowIN-1K Acc@1 ↑\uparrowADE20K mIoU ↑\uparrowCityscapes mIoU ↑\uparrow
MAE – standard i.i.d.77.02±0.02{77.02}_{\pm 0.02}25.9±0.1{{25.9}}_{\pm{0.1}}63.4±0.7{{63.4}}_{\pm{0.7}}81.12±0.10{{81.12}}_{\pm{0.10}}32.8±0.1{32.8}_{\pm 0.1}68.1±0.4{68.1}_{\pm 0.4}
MAE (streaming)77.02±0.09{77.02}_{\pm 0.09}24.1±0.2{24.1}_{\pm 0.2}60.6±0.6{60.6}_{\pm 0.6}80.05±0.04{80.05}_{\pm 0.04}26.6±0.2{26.6}_{\pm 0.2}58.9±0.5{58.9}_{\pm 0.5}
StreamMAE77.56±0.05{{77.56}}_{\pm{0.05}}25.8±0.3{25.8}_{\pm 0.3}63.3±0.6{63.3}_{\pm 0.6}81.12±0.00{{81.12}}_{\pm{0.00}}32.9±0.6{{32.9}}_{\pm{0.6}}69.5±0.4{{69.5}}_{\pm{0.4}}
Appendix
Table 13: Results across repeated downstream fine-tuning runs for fixed ViT-S/16 and ViT-B/16 checkpoints pretrained on WT++12h . We report mean and standard deviation over two seeds for ImageNet-1K and three seeds for ADE20K and Cityscapes.
MethodBatch sizeDataDropIN-1K Acc@1 ↑\uparrowCS (mIoU) ↑\uparrowADE (mIoU) ↑\uparrow
StreamMAE (ViT-B)B=512B=512✗81.773.636.4
B=256B=256✗81.873.335.4
B=128B=128✗81.472.234.2
Appendix
Table 14: Effect of reducing batch duration without DataDrop . With ViT-B/16 pretrained on WT++50h at 3.75 FPS, smaller batches correspond to shorter and more temporally correlated video windows, yet performance degrades only gradually.
MethodBatch sizeDataDrop (75%)IN-1K Acc@1 ↑\uparrowCS (mIoU) ↑\uparrowADE (mIoU) ↑\uparrow
MOCOv3 [ 10 ] baseline (ViT-S)B=2048B=2048✗66.351.718.6
B=2048B=2048✓68.752.019.4
MAE [ 26 ] baseline w/ Orthogonal-AdamW [ 22 ] (ViT-S)B=2048B=2048✗75.255.320.6
B=512B=512✗75.558.621.5
B=2048B=2048✓75.958.322.5
MAE [ 26 ] baseline (ViT-S)B=2048B=2048✗76.657.923.2
Appendix
Table 15: Effect of DataDrop on streaming baselines. We form a local window of B=2048B=2048 samples and backpropagate through a randomly selected 25%25\% subset. This generally improves over using the full window and matches or improves the B=512B=512 setting.
MethodWT durationIN-1K Acc@1 ↑\uparrowCS (mIoU) ↑\uparrowADE (mIoU) ↑\uparrow
StreamMAE (ViT-B)WT++12h +Budapest81.169.432.5
WT++25h +Budapest81.572.235.2
WT++50h81.973.835.6
Appendix
Table 16: Controlling for the final-video effect. Appending the same final video to shorter streams does not match the full WT++50h stream.
MethodMasking StrategyIN-1K Acc@1 ↑\uparrowCS (mIoU) ↑\uparrowADE (mIoU) ↑\uparrow
StreamMAE (ViT-B)Block Masking [ 55 ] (75%)80.968.132.8
I-JEPA [ 4 ] / Bootleg [ 33 ]80.466.931.7
Random Uniform [ 26 ] (75%)81.169.032.7
Appendix
Table 17: Effect of masking strategy for StreamMAE with ViT-B/16. Standard random uniform masking remains the strongest overall choice.
MethodAdamW β1\beta_{1}IN-1K Acc@1 ↑\uparrowCS (mIoU) ↑\uparrowADE (mIoU) ↑\uparrow
StreamMAE (ViT-B)0081.067.131.8
0.90.981.169.032.7
Appendix
Table 18: Effect of AdamW first-moment momentum for StreamMAE with ViT-B/16. Our standard setting β1=0.9\beta_{1}=0.9 performs better than deactivating momentum.
ComponentsDownstream performance
DataDropColor Jit. & Drop PathTwo-stage CroppingMotion-biased Crop SelectionCityscapes ↑\uparrowADE20K ↑\uparrowKITTI ↓\downarrow
––––60.823.24.307±0.046{4.307}_{\pm 0.046}
✓–––61.323.94.211±0.137{4.211}_{\pm 0.137}
✓✓––62.625.14.025±0.039{4.025}_{\pm 0.039}
✓✓✓–62.826.14.074±0.048{4.074}_{\pm 0.048}
✓–✓✓62.424.74.022±0.123{4.022}_{\pm 0.123}
Appendix
Table 19: Additional ablation of StreamMAE components with ViT-S/16 pretrained on WT++12h . The extra rows remove either DataDrop or regularization from the full method to isolate their roles.
MethodADE (mIoU) ↑\uparrowCS (mIoU) ↑\uparrowNYUv2 (RMSE) ↓\downarrowKITTI (RMSE) ↓\downarrow
StreamMAE w/o motion-biased selection33.971.50.621±0.004{\mathbf{0.621}}_{\pm\mathbf{0.004}}3.646±0.053{3.646}_{\pm 0.053}
StreamMAE (ours)34.572.20.621±0.001{\mathbf{0.621}}_{\pm\mathbf{0.001}}3.583±0.012{\mathbf{3.583}}_{\pm\mathbf{0.012}}
Appendix
Table 20: Ablation of motion-biased crop selection with ViT-B/16 pretrained on KrishnaCAM. Both variants use the same frames, pretraining budget, and remaining StreamMAE configuration.
MethodPretraining dataIN-1K Acc@1 ↑\uparrowCS (mIoU) ↑\uparrowADE (mIoU) ↑\uparrow
Standard i.i.d. MAE (ViT-S)ImageNet-1K78.565.928.5
StreamMAE (ViT-S)WT++95h78.468.029.5
Standard i.i.d. MAE (ViT-B)ImageNet-1K82.575.239.8
StreamMAE (ViT-B)WT++95h82.074.036.5
StreamMAE (ViT-B, avg. {60,100}%; see Table 7 )WT++95h81.975.337.3
Appendix
Table 21: Longer i.i.d. ImageNet reference. Unlike the main benchmarking tables, where ImageNet-1K i.i.d. MAE is iteration-matched to WT++12h , this table matches the training length of the WT++95h streaming runs, approximately 650k650\mathrm{k} iterations. StreamMAE trained on WT++95h remains competitive with this longer i.i.d. ImageNet reference, particularly for ViT-S and Cityscapes transfer.
Figure 7: Running average of cosine similarity between gradients from consecutive batches, measured on last-block MLP parameters every 10 steps.
Figure 8: Downstream performance versus the distance from standard i.i.d. MAE in mean gradient cosine similarity between consecutive batches throughout training. Extending Figure 4 , we include MAE trained on pre-shuffled & streaming WT++12h ( Table 12 ), StreamMAE with Orthogonal-AdamW, and streaming MAE using only motion-biased crop selection. All models use a ViT-S/16 backbone and are pretrained on WT++12h . All streaming variants use B=512B=512 without DataDrop .
Stream# VideosDuration 
WT++12h112.0hLondon stream
WT++25h1024.8hWT++12h + 9 videos from original WalkingTours [ 49 ]
WT++50h2650.0hWT++25h + 16 appended videos
WT++95h5894.5hFull WT++ collection
Appendix
Table 22: Construction of the ordered WT++ streams used in scaling experiments. Each stream is a prefix of the next one.
Stream stage# New videosCities / videos in training order
WT++12h1London
WT++25h9Venice, Amsterdam, Singapore, Istanbul, Bangkok, Stockholm, Kuala Lumpur, Zurich, Chiang Mai
WT++50h16Frankfurt, Phnom Penh, Sevilla, Hoi An, Prague, Chemnitz, Marmaris, Dubrovnik, Mostar, Gibraltar, Barcelona, Valencia, Rhodes, Helsinki, Copenhagen, Budapest
WT++95h32Vilnius, Danang, Klaipeda, Liege, Marbella, Monschau, Naples, Nessebar, Port de Soller, Sarajevo, Arcadia, Timisoara, Warsaw, Malacca, Florence, Cardiff, Georgetown, Luxembourg, Alicante, Ipoh, Belfast, Kaliningrad, Maastricht, Valletta, Vienna, Saigon, Cadiz, Kyiv, Oxford, Sihanoukville, Chongqing, Tokyo
Appendix
Table 23: Construction of the ordered WT++ streams. Each row lists the new videos introduced at that stage, in training order (all videos from earlier stages are retained).
ExperimentBackboneGPUsWall-clock time
StreamMAE pretraining on WT++12hViT-S/162×2\times H100∼\sim 7h
StreamMAE pretraining on WT++12hViT-B/162×2\times H100∼\sim 8h
MoCo v3 pretraining on WT++12hViT-S/162×2\times H100∼\sim 18h
DINO pretraining on WT++12hViT-S/162×2\times H100∼\sim 25h
ImageNet-1K fine-tuningViT-S/164×4\times H100∼\sim 4h
ImageNet-1K fine-tuningViT-B/164×4\times H100∼\sim 5h
Appendix
Table 24: Approximate compute requirements. Pretraining times are reported for WT++12h and exclude downstream fine-tuning.

Explore similar work

CardsList

  1. TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

    Sep 27, 2026Shih-Ying Yeh, Daniel Z. Kaplan, Xuehai Wang +3

    National Tsing Hua University · Comfy Org Research · Karolinska Institutet +2

  2. Image Classifiers are Efficient Self-Supervised Video Representation Learners

    Sep 30, 2026Owais Iqbal, Sudipta Sarkar, Shyam Marjit +3Vision TransformerImage Classification

    Indian Institute of Technology Kharagpur, India · Indian Institute of Science Bangalore, India · École de technologie supérieure Montreal, Canada

  3. Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streams

    May 29, 2026Jonathan Swinnen, Tinne TuytelaarsClass-Incremental LearningStreaming Video

    ESAT, KU Leuven, Leuven, Belgium