Given a set of procedural steps, Video Step Grounding (VSG) aims to identify which steps occur in a video and localize when they are performed. Existing methods are either offline, requiring access to the full video before making predictions, or online, but relying on fixed-length video segments producing noisy, over-segmented outputs. In this work, we propose Hierarchical Step Decomposition (HSD), a training-free framework for online VSG that leverages the semantic understanding capabilities of Large Multimodal Models. HSD addresses two central challenges in online grounding: constructing semantically coherent temporal segments from a streaming video, and bridging the semantic gap between coarse procedural step descriptions and visual observations. To construct elementary temporal units, HSD incrementally groups incoming segments into semantically consistent events, using an anchor caption that is continuously validated against the visual stream. Based on these events, we perform zero-shot procedural step classification and selectively refine uncertain predictions by decomposing candidate steps into finegrained substeps that capture more nuanced visual evidence. By operating on semantically coherent events rather than every fixed-length segment, HSD reduces redundant computation and produces more stable predictions. Experiments on HT-Step, Ego4D Goal-Step, and CrossTask show that HSD substantially outperforms state-of-the-art methods, overall achieving better step recognition, more accurate temporal localization, and faster inference.
Figures & tables
Figure 1: Top: previous works for Online VSG [ 45 ] predict procedural steps on fixed-length segments. Bottom: H ierarchical S tep D ecomposition (HSD) constructs semantically coherent temporal event segments and, if necessary, decomposes procedural steps into multiple substeps to mitigate the granularity mismatch between coarse procedural step descriptions and visual observations.
Figure 2: Overview of HSD. Given an input video V , the Online Semantic Event Construction module incrementally builds each event Ej from fixed-duration segments using a frozen LMM ΦLMM . The first segment of Ej is captioned as γanc , and each arriving segment is appended to Ej if its continuity score ϵsame exceeds ϵnew ; otherwise, Ej is terminated and a new event is initialized. For each completed event Ej , the Zero-Shot Event-to-Step Grounding module predicts a probability distribution pj over procedural steps. If its entropy Hj falls below threshold η , the most likely step is predicted. Otherwise, Hierarchical Step Refinement decomposes the top- k steps into substeps, re-evaluates Ej , aggregates substep scores into pjsub , and outputs the final prediction a^j .
Method
HT-Step mAP@IoU ↑
CrossTask mAP@IoU ↑
Ego4D Goal-Step R@1, IoU 0.3 ↑
@0.3
@0.5
@0.7
@[0.3–0.7]
@0.3
@0.5
@0.7
@[0.3–0.7]
Oracle
InternVL2.5-8B [ 6 ]
34.15
26.09
21.87
27.02
45.40
36.33
28.92
36.65
30.89
Qwen2.5-VL-7B [ 3 ]
41.24
32.83
28.04
33.76
48.26
38.89
31.26
39.23
32.07
Qwen3-VL-8B [ 2 ]
38.91
30.61
26.12
31.52
50.73
42.01
34.81
42.35
48.59
Dense video captioning (DVC)
Table 1: Comparison with state-of-the-art methods on HT-Step, CrossTask, and Ego4D Goal-Step. Methods marked operate online while methods marked are offline .
Figure 3: Inference-time comparison between BAGLM [ 45 ] , UniTime-7B [ 19 ] , Vid2Seq Boundary [ 43 ] and HSD on HT-Step and Ego4D Goal-Step using Qwen2.5-VL-7B.
Figure 4: Qualitative comparison between HSD and BAGLM.
OSEC
HSR
HT-Step mAP@IoU ↑
Ego4D Goal-Step R@1, IoU=0.3 ↑
@0.3
@0.5
@0.7
@[0.3–0.7]
✗
✗
18.10
8.84
3.53
9.87
8.08
✗
✓
20.89
10.31
3.95
11.44
8.22
✓
✗
23.65
12.56
5.73
13.72
17.48
✓
✓
25.92
13.43
5.74
14.68
19.51
Table 2: Component ablation study on HT-Step and Ego4D Goal-Step using Qwen2.5-VL-7B.
Trigger
HT-Step mAP@tIoU ↑
Ego4D Goal-Step R@1, IoU=0.3 ↑
@0.3
@0.5
@0.7
@[0.3–0.7]
None
23.65
12.56
5.73
13.72
17.48
Always
23.58
12.11
5.03
13.26
17.36
Random
18.05
9.05
3.85
10.64
17.09
Entropy
25.92
13.43
5.74
14.68
19.51
Oracle
41.85
23.05
10.22
21.09
33.26
Table 3: HSR activation strategy on HT-Step and Ego4D Goal-Step for Qwen2.5-VL-7B.
PSPW
Agg
HT-Step mAP@tIoU ↑
Ego4D Goal-Step R@1, IoU=0.3 ↑
@0.3
@0.5
@0.7
@[0.3–0.7]
✗
Sum
24.50
12.79
5.35
14.21
17.44
✓
Sum
25.03
12.93
5.59
14.52
19.02
✗
Max
25.05
13.30
5.52
14.62
17.53
✓
Max
25.92
13.43
5.74
14.68
19.51
Table 4: Ablation on Parent-Step Prior Weighting (PSPW) and Aggregation (Agg) on HT-Step and Ego4D Goal-Step using Qwen2.5-VL-7B.
Table 5: Performance comparison on CrossTask, HT-Step, and Ego4D Goal-Step.
Table 10
Temporal unit
#Units
@0.3
@0.5
@0.7
Avg.
Fixed windows
2 s (Table 2 )
190.7
18.10
8.84
3.53
9.87
4 s
95.1
22.96
10.63
3.80
12.05
8 s
47.3
23.81
10.47
3.12
11.87
Feature-change boundaries (DINOv2)
τ=0.6
19.9
17.79
9.59
4.59
10.49
Table 10: Alternatives to Online Semantic Event Construction (OSEC) on HT-Step with Qwen2.5-VL-7B and no Hierarchical Step Refinement. All rows share the same zero-shot event-to-step grounding and merging; only the temporal unit changes. #Units: average number of temporal units per video. Avg.: mAP averaged over tIoU ∈{0.3,0.4,0.5,0.6,0.7} . † Offline.
Figure 5: Qualitative results of HSD on videos from HT-Step ( Make Homemade Pocky ) and Ego4D Goal-Step ( Preparing leek ). Ground-truth step boundaries and predicted step probabilities per segment are shown for the baseline Qwen2.5-VL-7B, BAGLM with Qwen2.5-VL-7B, and HSD with the same backbone.
Figure 6: Qualitative results of HSD on two HT-Step videos: Make Turkey Meatloaf and Make Hungarian Goulash .
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Beijing Key Laboratory of Embodied Intelligence Computing, Beijing, China · Faculty of Computing, Harbin Institute of Technology, Weihai, China +1