Understanding instructional videos requires recognizing fine-grained actions and modeling their temporal relations, which remains challenging for current Video Foundation Models (VFMs). This difficulty stems from noisy web supervision and a pervasive "static bias", where models rely on objects rather than motion cues. To address this, we propose InstrAction, a pretraining framework for instructional videos' action-centric representations. We first introduce a data-driven strategy, which filters noisy captions and generates action-centric hard negatives to disentangle actions from objects during contrastive learning. At the visual feature level, an Action Perceiver extracts motion-relevant tokens from redundant video encodings. Beyond contrastive learning, we introduce two auxiliary objectives: Dynamic Time Warping alignment (DTW-Align) for modeling sequential temporal structure, and Masked Action Modeling (MAM) for strengthening cross-modal grounding. Finally, we introduce the InstrAct Bench to evaluate action-centric understanding, where our method consistently outperforms state-of-the-art VFMs on semantic reasoning, procedural logic, and fine-grained retrieval tasks.
Figures & tables
Figure 1: Video-language modeling paradigms. (a) Existing paradigms use trimmed clips with single atomic actions and perfectly aligned captions. (b) Instructional video setting involves sequential actions, loosely aligned descriptions, and non-instructional noise.
Figure 2: Overview of the LLM-assisted data curation pipeline. Our data-driven strategy consists of three stages: (I) filtering non-instructional noise from raw subtitles; (II) parsing the filtered captions to extract discrete action units into predefined structured formats; and (III) generating action-centric hard negatives. These include verb-altered negatives and order-swapped negatives.
Figure 3: Overview of the InstrAction framework. Our model builds on a video–text backbone and an Action Perceiver that extracts action-centric representations. The model is trained with three objectives: video–text contrastive learning, DTW-Align for temporal grounding, and Masked Action Modeling (MAM) for cross-modal action understanding.
Figure 4: Action Perceiver module.
Figure 5: Data Visualization. Top: Illustrative examples from our semantic and logic benchmarks. Bottom: Statistical breakdown of the dynamics benchmark, showcasing the diverse object pool and category proportions.
InstrAct-Semantic
InstrAct-Logic
Method
R@1 (%)
R@5 (%)
MR ( ↓ )
ACC (%)
ID Models (Zero-shot)
MIL-NCE Miech et al. (2020)
7.2
49.4
6.0
5.1
UNiVL Luo et al. (2020)
13.2
60.3
5.0
33.6
VideoPrism Zhao et al. (2024)
18.2
74.4
4.0
31.3
PerceptionLM Cho and others (2025)
16.5
63.2
4.0
31.4
Table 1: Performance on InstrAct-Semantic and InstrAct-Logic. Methods are grouped as In-domain (ID) and Out-of-domain (OOD). ↓ indicates lower is better.
Method
R@1
R@5
R@10
T2V
V2T
Mean
T2V
V2T
Mean
T2V
V2T
Mean
MIL-NCE Miech et al. (2020)
12.63
15.04
13.84
30.77
34.42
32.59
42.10
46.69
44.40
UniVL Luo et al. (2020)
13.77
15.91
14.84
32.67
36.80
34.73
43.89
49.14
46.51
PerceptionLM Cho and others (2025)
16.33
23.10
19.72
32.11
42.05
37.08
41.15
51.48
46.32
VideoPrism Zhao et al. (2024)
27.23
34.13
30.68
48.49
59.08
53.79
58.36
70.00
64.18
Ours
33.86
35.19
34.53
54.73
59.52
57.13
63.13
69.71
66.42
Table 2: Overall performance on the InstrAct-Dynamics benchmark. Weighted average Recall@K across 120 objects (16,326 clips).
Figure 6: Qualitative results of text-to-video retrieval on InstrAct-Dynamics pools.
Backbone
Method
InstrAct-Semantic
InstrAct-Logic
R@1 (%)
R@5 (%)
MR ( ↓ )
ACC (%)
Backbone
7.2
49.4
6.0
5.1
MIL-NCE Miech et al. (2020)
+ HNs only
32.4
82.2
3.0
6.1
+ InstrAction
32.9
83.3
2.0
4.7
Backbone
18.4
63.2
4.0
24.5
CLIP-ViP Xue et al. (2023)
+ HNs only
38.5
86.7
2.0
30.2
Table 3: Generalization across different Video Foundation Models. We evaluate the performance gains of applying solely our data-driven augmentation (+ HNs only) versus deploying our complete InstrAction framework which incorporates both data- and model-driven approaches (+ InstrAct) across four distinct backbones.
Figure 7: Qualitative comparison of DTW alignment heatmaps. Left subplots show Baseline (Raw Frames → Actions), and Right subplots show (Latents → Actions).
Visual Representation
Cost ↓
Raw Embeddings
0.97
Action Latents
0.91
Figure 8: Quantitative comparison of alignment cost (lower is better).
Components
InstrAct-Semantic
Logic
HN
Action Perceiver
Align.
R@1
R@5
MR ( ↓ )
ACC
✓
–
–
40.7
89.4
2.0
37.2
✓
✓
MAM
43.7
90.0
2.0
39.4
✓
✓
DTW-Align
38.7
86.5
2.0
40.2
✓
✓
Both
45.1
90.8
2.0
44.7
Table 4: Ablation Study on InstrAct Bench. We evaluate the impact of Hard Negatives (HN), and 2 proposed constrains. ↓ indicates lower is better.
Video understanding develops methods for recognizing actions, tracking events, grounding evidence in time, and reasoning across visual and audio streams.