Our objective is to enhance VLMs ability to distinguish the roles of the left and right hands in egocentric actions. We introduce sub-action detection, a task that requires models to explicitly recognise the left- and right-hand roles and their manipulated objects within an ongoing high-level action. To address this challenge, we propose BiScopeNet, a framework that augments existing video-LLMs without modifying their vision encoders or language models. BiScopeNet incorporates two components: FocusAlign, a gated text-guided attention-based module that selectively aligns video features with textual context to emphasise action-relevant cues, and MetaLoss, a masked, object-weighted loss that prioritises visually grounded terms. To supervise sub-action detection, we introduce BiScopeCap-100K, a large-scale video-level dataset curated from existing egocentric datasets, providing dense captions that jointly describe high-level actions, left/right-hand roles, and manipulated objects, enriched with spatial metadata for grounding. Across the evaluated Video-LLM backbones, BiScopeNet improves action, hand-role, and object recognition over the corresponding supervised baselines, achieving the strongest aggregate performance among the evaluated configurations on BiScopeCap.
Figures & tables
Figure 1: Comparison of BiScopeNet and baseline models, showing how the new data incorporation approach, along with the proposed projection and loss mechanisms, improves hand sub-action recognition, overall action understanding, and object grounding in generated captions.
Figure 2: Overview of BiScopeNet. Video frames are encoded into visual tokens and processed by our FocusAlign, which integrates text-guided cross-attention, adaptive gating, and MLP refinement to prioritise action-relevant patches. The resulting representation is projected into the LLM space to generate fine-grained captions that capture hand interactions underlying the main action.
Dataset
Action
Hand Role
Spatial
Coarse-grained
Fine-grained
Domain
# Train
# Test
Label
Labels
Metadata
Captions
Captions
Source
Samples
Samples
Epic-Kitchens / Visor
✓
✗
✗
✗
✗
Kitchen / Daily Tasks
31,432
3,729
Ego4D
✓
✗
✗
✗
✗
Diverse Activities
22,916
3,304
HOI-QA
✓
✗
✓
✗
✓
Kitchen / Daily Tasks
–
–
HD-Epic
✓
✓
✗
✗
✓
Kitchen / Daily Tasks
38,520
2,846
BiScopeCap (ours)
✓
✓
✓
✓
✓
Diverse Activities
92,868
9,879
Table 1: Comparison between existing egocentric datasets and our proposed BiScopeCap , along with the distribution of samples across its constituent subsets.
Method
# Samples
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
GPT Scoring
200
44.5
31.0
36.0
47.0
Human Scoring
200
42.0
30.3
35.6
51.0
Table 2: Comparison between LLM-as-a-judge scores and human evaluation over 200 predicted captions.
Method
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
BiScopeNet (avg. 3 runs)
45.3
44.7
40.6
58.9
Standard Deviation
± 0.2
± 0.5
± 0.6
± 0.07
Table 3: BiScopeNet stability results across three independent inference and scoring runs on NVILA backbone.
Method
Vision Encoder
Backbone
Trained
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
# Unstruct ↓
SmolVLM [ 33 ]
SigLIP [ 58 ]
-
✗
10.4
11.8
14.6
27.6
4505
Qwen 2.5-VL [ 3 ]
-
-
✗
25.6
29.9
37.0
54.0
1
Qwen 3.5 [ 48 ]
-
-
✗
26.0
32.2
40.6
53.3
6
Intern VL 2.5 [ 7 ]
InternViT [ 8 ]
-
✗
24.8
27.4
37.0
51.5
5
NVILA [ 29 ]
SigLIP
-
✗
14.1
17.2
28.7
39.7
0
NVILA [ 29 ]
SigLIP
-
✓
44.0
43.0
39.1
58.0
0
Table 4: Comparison of BiScopeNet with state-of-the-art VLMs on BiScopeCap. Blue denote the best results.
Table 6: Ablation study of BiScopeNet components using NVILA backbone [ 29 ] on BiScopeCap. The proposed FocusAlign and MetaLoss consistently improve all metrics, demonstrating complementary benefits for sub-action understanding. Blue indicates the best results.
Method
Component
Vision Encoder
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
# Unstruct ↓
VideoChatGPT
Linear + Cross Entropy
CLIP
12.8
10.4
5.9
27.3
374
VideoChatGPT
MLP + Cross Entropy
CLIP
16.8
15.5
12.8
26.4
437
BiScopeNet
FocusAlign + Cross Entropy
CLIP
13.9
10.6
12.9
18.8
196
BiScopeNet
Linear + MetaLoss
CLIP
19.6
19.0
25.8
40.4
0
BiScopeNet
FocusAlign + MetaLoss
CLIP
32.5
18.4
25.8
46.4
5
BiScopeNet
FocusAlign + MetaLoss + CLS
CLIP
30.7
19.4
29.6
47.9
7
Table 7: Ablation study of BiScopeNet components using VideoChatGPT backbone [ 31 ] on the Epic subset of BiScopeCap. The proposed FocusAlign and MetaLoss consistently deliver the best performance, highlighting their complementary benefits for sub-action understanding. Blue indicates the best results.
Method
Trained
Vision Encoder
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
# Unstruct ↓
Ego4D
SmolVLM [ 33 ]
✗
SigLIP
8.9
12.0
16.5
25.8
1256
NVILA [ 29 ]
✗
SigLIP
13.0
16.7
30.9
39.7
0
VideoChatGPT [ 31 ]
✓
CLIP
7.2
10.1
12.3
24.0
563
BiScopeNet (Ours)
✓
DINOv3
22.1
38.7
39.6
42.1
46
Epic+Ego4D
SmolVLM [ 33 ]
✗
SigLIP
13.1
15.6
18.9
29.6
2785
NVILA [ 29 ]
✗
SigLIP
17.7
21.5
34.9
43.1
0
Table 8: Comparison of BiScopeNet using VideoChatGPT [ 31 ] as backbone with state-of-the-art VLMs across Ego4D , Epic+Ego4D , and HD-Epic subsets of BiScopeCap. # Unstruct counts are over the corresponding subset test split, not the full BiScopeCap test set. Blue denote the best results.
Method
Vision Encoder
CLS
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
# Unstruct ↓
VideoChatGPT
CLIP
✗
12.8
10.4
5.9
27.3
374
BiScopeNet
CLIP
✗
32.5
18.4
25.8
46.4
5
BiScopeNet
CLIP
✓
30.7
19.4
29.6
47.9
7
BiScopeNet
DINOv3
✗
33.1
17.6
36.6
48.0
5
BiScopeNet
DINOv3
✓
28.2
23.0
38.2
49.4
13
BiScopeNet
SigLIP2
✗
32.4
21.6
31.8
47.0
5
Table 9: Ablation study of BiScopeNet integrated into the VideoChatGPT [ 31 ] backbone with different vision encoders (CLIP, DINOv3, and SigLIP2) and the presence or absence of the CLS token on the Epic subset of BiScopeCap. BiScopeNet consistently improves performance across all encoders. SigLIP2+CLS achieves the highest overall and object-level accuracy, while DINOv3 shows superior sensitivity to hand-specific cues, demonstrating the adaptability of BiScopeNet to diverse vision backbones. Blue highlights the best results.
Figure 3: Qualitative comparison of visual understanding across methods.
Figure 4: Qualitative comparison of output structure across methods.
Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.
Fine-grained action understanding requires vision-language models to ground what an agent does, how it acts, and why it acts in spatio-temporal evidence. However, richer supervision can backfire: fine-tuned models may generate detailed action descriptions grounded in static scenes and salient objects rather than motion, a failure we call Environment-Grounded Action Hallucination (EGAH). To mitigate EGAH, we propose Staged Attention Rectification and Reinforcement (STARR), an action-centric video-text alignment framework that breaks scene-action shortcuts through Action-Environment Dissociation and reinforces action-focused grounding with an Attention Consistency Reward within Group Relative Policy Optimization. We further construct FineActionR&R, combining temporally localized captions with hierarchical and temporal-causal reasoning data. Built on Qwen2.5-VL-7B, FineActionVLM-R1 improves average MotionBench accuracy by 3.01 percentage points over its base model and achieves 91.50% on HAU-MCQ-Bench. Qualitative comparisons further indicate more faithful fine-grained action descriptions with fewer environment-induced hallucinations.
Zhiliang Zhu, Jiazheng Huang, Dawei Zhao
Harbin Engineering University, China · South China Normal University, China
Fine-grained action recognition in egocentric video is challenging for Vision-Language Models (VLMs): actions often differ only in small visual cues, and a single model tends to be biased toward a subset of these cues. We propose Divide, Deliberate, Decide, a fully-local, zero-shot multi-agent framework in which (i) a VLM orchestrator chunks the video and proposes a top-k candidate label list per segment, (ii) an ensemble of heterogeneous VLM specialists, drawn from different open model families, engages in a structured deliberation that includes a peer-consultation round of questions, and (iii) agent rankings are aggregated with a Borda count and the orchestrator re-ranks its own prediction in light of the specialists' evidence. The entire pipeline runs locally with no fine-tuning. Experiments show that our method positively improves zero-shot action recognition performance over the baseline, highlighting the influence of a heterogeneous deliberation step, showing that the gain stems from decorrelated model priors rather than from additional compute.
Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions, a task where general-purpose Vision-Language Models (VLMs) often struggle when operating directly on raw pixels. We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs. In a multi-stage prompting pipeline, we first generate dense natural language narratives over short temporal windows as a semantic bottleneck, then formalize them into structured, open-vocabulary graph representations. On the EGTEA and Epic-Kitchens-100 datasets, the symbolic representation unlocks efficient in-context learning: few-shot graph demonstrations yield substantial accuracy gains over zero-shot frame-based inference. Even in the zero-shot setting, graph-based reasoning remains competitive with pixel-based inference despite potential pretraining contamination favoring the latter. Across 11 open-weight VLMs from 6 model families ranging from 2B to 235B parameters, our findings indicate that current VLMs are more effective as symbolic reasoners than as direct visual observers. By projecting video into the language domain, we provide a scalable, fine-tuning-free alternative to end-to-end approaches that better leverages these models' latent reasoning strengths. The code will be made public.
Bessie Dominguez-Dager, Francisco Gomez-Donoso, Miguel Cazorla +3