Not All Layers Are Equal in Image-to-Video Transfer
Authors: Thinesh Thiyakesan Ponbagavathi, Constantin Seibold, Alina Roitberg
Organizations: University of Stuttgart · Intelligent Assistive Systems Lab, University of Hildesheim · University Hospital Heidelberg, Diagnostic and Interventional Radiology · Multimodal Machine Learning and Assistive Computing Lab, University of Münster
Parameter-efficient image-to-video transfer adapts frozen image foundation models to video via lightweight trainable adapters. Existing methods insert the same adapter at every transformer layer, implicitly assuming all layers require equal temporal modeling. A layer-wise sensitivity analysis across four video-native model families and two pretraining paradigms reveals that this assumption is wrong: temporal sensitivity increases with depth, and the pretraining paradigm determines its growth profile. Vision-only models exhibit a gradual rise, while vision-language models show early suppression followed by late recovery. This finding motivates STRAP, a training-free adapter placement rule that derives a spatial-to-temporal transition boundary from a paradigm-matched video model. Spatial adapters are assigned before this boundary and temporal adapters after it, without modifying the adapter architecture itself. On four fine-grained action recognition benchmarks, STRAP improves accuracy by up to 11.8% over uniform placement, reduces trainable parameters by 29 to 48%, and lowers inference time by up to 10%. On standard benchmarks, it matches accuracy with fewer parameters. Code is available at https://github.com/th-nesh/STRAP.
Figures & tables
Figure 1: Existing image-to-video adapters place the same temporal adapter across all layers. Strap instead uses a sensitivity-derived, backbone-specific boundary to place spatial adapters in early layers and temporal adapters in later layers.
Figure 2: Sensitivity Metric Pipeline. Spatially and temporally perturbed versions of a video clip are passed through a frozen video-native model together with the original clip. Layer-wise cosine distances between mean-pooled patch-token embeddings measure each layer’s response to spatial and temporal perturbations, yielding the Temporal Sensitivity Ratio R(l) .
Figure 3: Temporal sensitivity profiles are depth-dependent and paradigm-determined. R(l) increases with depth across all models (Finding 1). Vision-only models show gradual emergence; vision-language models exhibit early suppression followed by a late-layer rise (Finding 2), consistent with the influence of cross-modal alignment. Profiles are stable across scales and datasets.
Params
Inf. Time
Top-1 Accuracy (%) ↑
Method
Strategy
(M)
(ms) †
Diving
Drive&Act
IKEA
HRI-30
Fully Fine-tuned Frameworks
ORViT [ 31 ]
-
160
-
88.0
-
-
-
VideoSwin-B [ 32 ]
-
88
-
-
72.9
72.6
-
UniformerV2 [ 33 ]
-
115
-
-
76.7
-
-
SlowOnly [ 34 ]
-
32
-
-
-
-
86.6
Table 1: Fine-grained action recognition. STRAP improves accuracy (up to +11.8%) while reducing trainable parameters by up to 48% and inference time by up to 10%. Gains are largest on temporally demanding datasets (D&A, IKEA, HRI) and consistent across methods and backbones.
Drive&Act
IKEA-ASM
HRI-30
Method
Strat.
Sym
N-S
All
Sym
N-S
All
Sym
N-S
All
ST-Adapter
Uni.
66.4
83.4
75.2
68.5
73.9
70.5
81.5
95.3
85.5
ST-Adapter
Strap
78.0
85.5
81.9
80.3
76.2
77.6
85.4
96.8
89.2
Frame2Freq
Uni.
77.1
85.5
81.5
81.0
74.2
76.4
85.6
98.9
89.8
Strap
79.6
87.1
83.5
86.8
76.4
79.8
88.2
97.2
91.2
Uni.
68.4
85.0
77.7
75.3
74.6
74.9
78.4
84.6
80.5
Table 2: Results for symmetric and non-symmetric actions. Sym and N-S denote accuracy on symmetric and non-symmetric action subsets. Our method has the largest gains for symmetric actions (e.g., opening vs. closing ), which require distinguishing temporal order and direction, precisely the structure targeted by late-layer temporal adapters.
Figure 4: Ablation of spatial-temporal adapter allocation across layers. We vary the number of spatial (S) and temporal (T) adapters under varying depth.
Table 7
SSv2
K400
Method
Strategy
Params (M)
Top-1 (%)
Params (M)
Top-1 (%)
Fully Fine-tuned Frameworks
Video SWIN [ 32 ]
-
88.1
69.6
88.1
80.6
ViT-B (finetuned)
-
86.1
69.3
86.1
81.0
ILA [ 14 ]
-
-
66.8
-
81.6
STAN-self [ 12 ]
-
-
69.5
-
84.2
Table 6: Standard (non-fine-grained) video classification results. Strap reduces trainable parameters while slightly improving accuracy across most settings.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Impact of spatial perturbation choice on R(l) . Leave-one-out analysis removing each spatial perturbation in turn: random crop, spatial blur, and color jitter. The depth-wise profile and paradigm-specific signatures remain consistent across all variants for both VideoMAE and ViCLIP.
Figure 6: Impact of temporal perturbation choice on R(l) . Leave-one-out analysis removing each temporal perturbation in turn: frame shuffle, frame reversal, and frame drop. The depth-wise profile and paradigm-specific signatures remain stable across all variants for both VideoMAE and ViCLIP.
Figure 7: Layer-wise temporal sensitivity R(l) of four additional video foundation models. R(l) increases with depth, and its growth profile depends on the pretraining paradigm.
Backbone
Strategy
Params (M)
Drive&Act
IKEA-ASM
HRI-30
DINOv2 (ST-Adapter)
All Spatial
1.1
73.7
70.2
68.5
10S/2T
2.2
80.6
75.9
80.6
8S/4T
3.1
81.1
77.1
87.9
6S/6T †
4.2
81.9
77.6
89.2
4S/8T
5.2
80.9
76.5
88.0
2S/10T
6.1
82.6
75.3
91.1
Appendix
Table 7: Ablation of spatial–temporal adapter allocation across layers for ST-Adapter. We vary the number of spatial (S) and temporal (T) adapters under varying depth (parameter count varies accordingly). Bold indicates the best result per column.
Backbone
Strategy
Params (M)
Drive&Act
IKEA-ASM
HRI-30
DINOv2 (Frame2Freq)
All Spatial
1.1
73.7
70.2
68.5
10S/2T
2.2
82.4
77.7
90.1
8S/4T
3.1
83.1
75.1
90.8
6S/6T †
4.2
83.5
79.8
91.2
4S/8T
5.2
82.3
78.6
89.5
2S/10T
6.1
82.4
78.6
84.6
Appendix
Table 8: Ablation of spatial–temporal adapter allocation across layers for Frame2Freq. We vary the number of spatial (S) and temporal (T) adapters under varying depth (parameter count varies accordingly). Bold indicates the best result per column.
Backbone
Strategy
Params (M)
Drive&Act
IKEA-ASM
HRI-30
DINOv2 (M2-CLIP)
All Spatial
1.1
73.7
70.2
68.5
10S/2T
3.4
81.8
76.9
88.2
8S/4T
5.6
82.5
78.7
91.0
6S/6T †
7.7
82.9
79.6
92.3
4S/8T
9.9
82.3
79.1
90.2
2S/10T
12.1
82.5
77.4
91.6
Appendix
Table 9: Ablation of spatial–temporal adapter allocation across layers for M2-CLIP. We vary the number of spatial (S) and temporal (T) adapters under varying depth (parameter count varies accordingly). Bold indicates the best result per column.
SSv2-Small
HMDB-51
UCF-101
Method
Strategy
Params(M)
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
State-of-the-Art Methods
TATs [ 48 ]
-
18.1
47.9
64.4
60.0
77.0
92.0
95.5
Trokens [ 47 ]
-
17.4
53.4
68.9
69.8
82.3
94.0
97.9
D2-ST Adapter [ 44 ]
-
7.3
55.0
69.3
77.1
88.2
96.4
99.1
Parameter Efficient Fine-tuning Frameworks
Appendix
Table 10: Few-shot action recognition. Strap improves over uniform placement in all 18 settings with no degradation, while reducing parameters by 29-33%. State-of-the-art methods are shown for reference; Strap achieves competitive performance with substantially fewer parameters.
Drive&Act
IKEA-ASM
HRI-30
Method
Strategy
Sym
N-Sym
Top-1
Sym
N-Sym
Top-1
Sym
N-Sym
Top-1
DINOv2-B/14 backbone (6S/6T, derived from VideoMAE )
ST-Adapter [ 1 ]
Uniform
66.4
83.4
75.2
68.5
73.9
70.5
81.5
95.3
85.5
Strap
78.0
85.5
81.9
80.3
76.2
77.6
85.4
96.8
89.2
Frame2Freq [ 3 ]
Uniform
77.1
85.5
81.5
81.0
74.2
76.4
85.6
98.9
89.8
Strap
79.6
87.1
83.5
86.8
76.4
79.8
88.2
97.2
91.2
Appendix
Table 11: Fine-grained action recognition with symmetric and non-symmetric breakdown. Sym and N-Sym denote accuracy on symmetric and non-symmetric action subsets, respectively; Top-1 is overall accuracy. STRAP’s largest gains concentrate on symmetric actions, which require distinguishing temporal order and direction -precisely the structure targeted by late-layer temporal adapters.
mAP @ tIoU
Config
Avg
0.3
0.4
0.5
0.6
0.7
Params
Spatial-only [ 28 ]
69.03
84.76
79.93
72.34
61.53
46.62
1.2M
Uniform [ 50 ]
70.36
85.58
81.32
73.96
61.95
49.00
4.0M
Strap
70.75
86.06
80.88
74.41
63.05
49.34
3.1M
Appendix
Table 12: Temporal action detection on THUMOS14 (VideoMAE, OpenTAD). Average mAP and mAP at each tIoU threshold. Best result per accuracy column in bold.
Institute for Artificial Intelligence, University of Stuttgart, Germany · Intelligent Assistive Systems Lab, University of Hildesheim · University Hospital Heidelberg, Diagnostic and Interventional Radiology