cs.CVNov 8, 2026

Not All Layers Are Equal in Image-to-Video Transfer

Authors: Thinesh Thiyakesan Ponbagavathi, Constantin Seibold, Alina Roitberg

Organizations: University of Stuttgart · Intelligent Assistive Systems Lab, University of Hildesheim · University Hospital Heidelberg, Diagnostic and Interventional Radiology · Multimodal Machine Learning and Assistive Computing Lab, University of Münster

Abstract

Parameter-efficient image-to-video transfer adapts frozen image foundation models to video via lightweight trainable adapters. Existing methods insert the same adapter at every transformer layer, implicitly assuming all layers require equal temporal modeling. A layer-wise sensitivity analysis across four video-native model families and two pretraining paradigms reveals that this assumption is wrong: temporal sensitivity increases with depth, and the pretraining paradigm determines its growth profile. Vision-only models exhibit a gradual rise, while vision-language models show early suppression followed by late recovery. This finding motivates STRAP, a training-free adapter placement rule that derives a spatial-to-temporal transition boundary from a paradigm-matched video model. Spatial adapters are assigned before this boundary and temporal adapters after it, without modifying the adapter architecture itself. On four fine-grained action recognition benchmarks, STRAP improves accuracy by up to 11.8% over uniform placement, reduces trainable parameters by 29 to 48%, and lowers inference time by up to 10%. On standard benchmarks, it matches accuracy with fewer parameters. Code is available at https://github.com/th-nesh/STRAP.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Frame2Freq: Spectral Adapters for Fine-Grained Video Understanding

    Feb 21, 2026Thinesh Thiyakesan Ponbagavathi, Constantin Seibold, Alina Roitbergfine-grained action understandingVideo Action Recognition

  2. Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models

    May 18, 2026Svetlana Orlova, Niccolò Cavagnero, Gijs DubbelmanStreaming Video UnderstandingVision Foundation Model Adaptation

  3. SV-TAD: Native Sparse Convs for Efficient Temporal Action Detection

    Oct 8, 2026Ricardo Pizarro, Roberto Valle, José M. Buenaposada +2Temporal Action DetectionEfficient ViTs