Given a set of procedural steps, Video Step Grounding (VSG) aims to identify which steps occur in a video and localize when they are performed. Existing methods are either offline, requiring access to the full video before making predictions, or online, but relying on fixed-length video segments producing noisy, over-segmented outputs. In this work, we propose Hierarchical Step Decomposition (HSD), a training-free framework for online VSG that leverages the semantic understanding capabilities of Large Multimodal Models. HSD addresses two central challenges in online grounding: constructing semantically coherent temporal segments from a streaming video, and bridging the semantic gap between coarse procedural step descriptions and visual observations. To construct elementary temporal units, HSD incrementally groups incoming segments into semantically consistent events, using an anchor caption that is continuously validated against the visual stream. Based on these events, we perform zero-shot procedural step classification and selectively refine uncertain predictions by decomposing candidate steps into finegrained substeps that capture more nuanced visual evidence. By operating on semantically coherent events rather than every fixed-length segment, HSD reduces redundant computation and produces more stable predictions. Experiments on HT-Step, Ego4D Goal-Step, and CrossTask show that HSD substantially outperforms state-of-the-art methods, overall achieving better step recognition, more accurate temporal localization, and faster inference.
Figures & tables
Figure 1: Top: previous works for Online VSG [ 45 ] predict procedural steps on fixed-length segments. Bottom: H ierarchical S tep D ecomposition (HSD) constructs semantically coherent temporal event segments and, if necessary, decomposes procedural steps into multiple substeps to mitigate the granularity mismatch between coarse procedural step descriptions and visual observations.
Figure 2: Overview of HSD. Given an input video V , the Online Semantic Event Construction module incrementally builds each event Ej from fixed-duration segments using a frozen LMM ΦLMM . The first segment of Ej is captioned as γanc , and each arriving segment is appended to Ej if its continuity score ϵsame exceeds ϵnew ; otherwise, Ej is terminated and a new event is initialized. For each completed event Ej , the Zero-Shot Event-to-Step Grounding module predicts a probability distribution pj over procedural steps. If its entropy Hj falls below threshold η , the most likely step is predicted. Otherwise, Hierarchical Step Refinement decomposes the top- k steps into substeps, re-evaluates Ej , aggregates substep scores into pjsub , and outputs the final prediction a^j .
Method
HT-Step mAP@IoU ↑
CrossTask mAP@IoU ↑
Ego4D Goal-Step R@1, IoU 0.3 ↑
@0.3
@0.5
@0.7
@[0.3–0.7]
@0.3
@0.5
@0.7
@[0.3–0.7]
Oracle
InternVL2.5-8B [ 6 ]
34.15
26.09
21.87
27.02
45.40
36.33
28.92
36.65
30.89
Qwen2.5-VL-7B [ 3 ]
41.24
32.83
28.04
33.76
48.26
38.89
31.26
39.23
32.07
Qwen3-VL-8B [ 2 ]
38.91
30.61
26.12
31.52
50.73
42.01
34.81
42.35
48.59
Dense video captioning (DVC)
Table 1: Comparison with state-of-the-art methods on HT-Step, CrossTask, and Ego4D Goal-Step. Methods marked operate online while methods marked are offline .
Figure 3: Inference-time comparison between BAGLM [ 45 ] , UniTime-7B [ 19 ] , Vid2Seq Boundary [ 43 ] and HSD on HT-Step and Ego4D Goal-Step using Qwen2.5-VL-7B.
Figure 4: Qualitative comparison between HSD and BAGLM.
OSEC
HSR
HT-Step mAP@IoU ↑
Ego4D Goal-Step R@1, IoU=0.3 ↑
@0.3
@0.5
@0.7
@[0.3–0.7]
✗
✗
18.10
8.84
3.53
9.87
8.08
✗
✓
20.89
10.31
3.95
11.44
8.22
✓
✗
23.65
12.56
5.73
13.72
17.48
✓
✓
25.92
13.43
5.74
14.68
19.51
Table 2: Component ablation study on HT-Step and Ego4D Goal-Step using Qwen2.5-VL-7B.
Trigger
HT-Step mAP@tIoU ↑
Ego4D Goal-Step R@1, IoU=0.3 ↑
@0.3
@0.5
@0.7
@[0.3–0.7]
None
23.65
12.56
5.73
13.72
17.48
Always
23.58
12.11
5.03
13.26
17.36
Random
18.05
9.05
3.85
10.64
17.09
Entropy
25.92
13.43
5.74
14.68
19.51
Oracle
41.85
23.05
10.22
21.09
33.26
Table 3: HSR activation strategy on HT-Step and Ego4D Goal-Step for Qwen2.5-VL-7B.
PSPW
Agg
HT-Step mAP@tIoU ↑
Ego4D Goal-Step R@1, IoU=0.3 ↑
@0.3
@0.5
@0.7
@[0.3–0.7]
✗
Sum
24.50
12.79
5.35
14.21
17.44
✓
Sum
25.03
12.93
5.59
14.52
19.02
✗
Max
25.05
13.30
5.52
14.62
17.53
✓
Max
25.92
13.43
5.74
14.68
19.51
Table 4: Ablation on Parent-Step Prior Weighting (PSPW) and Aggregation (Agg) on HT-Step and Ego4D Goal-Step using Qwen2.5-VL-7B.
Table 5: Performance comparison on CrossTask, HT-Step, and Ego4D Goal-Step.
Table 10
Temporal unit
#Units
@0.3
@0.5
@0.7
Avg.
Fixed windows
2 s (Table 2 )
190.7
18.10
8.84
3.53
9.87
4 s
95.1
22.96
10.63
3.80
12.05
8 s
47.3
23.81
10.47
3.12
11.87
Feature-change boundaries (DINOv2)
τ=0.6
19.9
17.79
9.59
4.59
10.49
Table 10: Alternatives to Online Semantic Event Construction (OSEC) on HT-Step with Qwen2.5-VL-7B and no Hierarchical Step Refinement. All rows share the same zero-shot event-to-step grounding and merging; only the temporal unit changes. #Units: average number of temporal units per video. Avg.: mAP averaged over tIoU ∈{0.3,0.4,0.5,0.6,0.7} . † Offline.
Figure 5: Qualitative results of HSD on videos from HT-Step ( Make Homemade Pocky ) and Ego4D Goal-Step ( Preparing leek ). Ground-truth step boundaries and predicted step probabilities per segment are shown for the baseline Qwen2.5-VL-7B, BAGLM with Qwen2.5-VL-7B, and HSD with the same backbone.
Figure 6: Qualitative results of HSD on two HT-Step videos: Make Turkey Meatloaf and Make Hungarian Goulash .
Procedural activities follow well-defined structures: whether we consider a cooking recipe or a mechanic repairing a car, these activities naturally decompose in a hierarchy of steps and sub-steps. Traditional approaches for step grounding require extensive annotations and scale poorly. Instead, we argue that such hierarchical structure can emerge naturally from uncurated videos of human activities through recurring patterns of co-occurring actions and activities. Our approach builds on HiERO, a weakly-supervised representation learning approach that maps close in the feature space actions that are functionally related to each other, leveraging only fine-grained action-level narrations. In this feature space, procedure steps can be detected by a simple clustering, with no additional task-specific fine-tuning. For the Ego4D Step Grounding challenge, we augment this approach by ensuring fine and coarse level agreement in step assignments, enforcing strict temporal monotonicity of the grounded steps and post-processing the detected steps to reduce the impact of noisy predictions. We call this approach HiERO-StepG and it achieves 56.27 % on the R@1 (IoU = 0.3) metric on the global leaderboard at submission time, ranking second while being completely zero-shot and not requiring procedure-specific annotations. Project page: https://github.com/andreazenotto/HiERO-StepG.
Andrea Zenotto, Simone Alberto Peirone, Francesca Pistilli +1
Translating continuous, noisy egocentric video streams into discrete, temporally ordered action steps is fraught with visual challenges. Heavy ego-motion, transient occlusions, and the high intra-class variability of unscripted human-object interactions cause standard frame-level online temporal models to struggle, often resulting in severe over-segmentation and structural collapse. To bridge the gap between unstable low-level perception and high-level procedural logic, we present VidParse, an online, training-free framework that treats activity understanding as a graph-constrained inference problem. Rather than relying on learned temporal filters, we dynamically identify semantic transitions using a temporal similarity matrix over manipulation-anchored features, which are extracted from frozen foundation models to prioritize foreground hand-object interactions. A beam search decoder then leverages an induced procedural task graph to explicitly enforce valid action transitions and prune impossible trajectories. By anchoring robust visual segments to hard procedural constraints, our approach preserves long-range state transitions and achieves up to a 10x improvement in complex multi-step parsing accuracy over strong online baselines, all without requiring a single gradient update.
Current Video-LLM approaches for Video Temporal Grounding (VTG) typically rely on direct timestamp generation from an unstructured visual-token stream, often leading to brittle numerics and inconsistent boundaries. To address this, we propose Foresee-to-Ground (F2G), a framework that reformulates VTG as a verifiable Identify-then-Measure problem. F2G integrates Predictive Temporal Perception with Evidence-Driven Reasoning: it learns boundary-sensitive temporal representations to build a video-wide evidence pool of candidate event segments, and exposes these segments to the LLM as citable evidence units that bind boundary prediction to explicit event hypotheses. By decoupling event identification from precise boundary measurement, F2G stabilizes grounding and makes predictions verifiable. Extensive experiments demonstrate that F2G consistently improves grounding accuracy across diverse benchmarks, transfers robustly across different Video-LLM backbones, and preserves general video understanding capabilities.
Zelin Zheng, Xinyan Liu, Ruixin Li +4
School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China · Beijing Key Laboratory of Embodied Intelligence Computing, Beijing, China · Faculty of Computing, Harbin Institute of Technology, Weihai, China +1