Our objective is to enhance VLMs ability to distinguish the roles of the left and right hands in egocentric actions. We introduce sub-action detection, a task that requires models to explicitly recognise the left- and right-hand roles and their manipulated objects within an ongoing high-level action. To address this challenge, we propose BiScopeNet, a framework that augments existing video-LLMs without modifying their vision encoders or language models. BiScopeNet incorporates two components: FocusAlign, a gated text-guided attention-based module that selectively aligns video features with textual context to emphasise action-relevant cues, and MetaLoss, a masked, object-weighted loss that prioritises visually grounded terms. To supervise sub-action detection, we introduce BiScopeCap-100K, a large-scale video-level dataset curated from existing egocentric datasets, providing dense captions that jointly describe high-level actions, left/right-hand roles, and manipulated objects, enriched with spatial metadata for grounding. Across the evaluated Video-LLM backbones, BiScopeNet improves action, hand-role, and object recognition over the corresponding supervised baselines, achieving the strongest aggregate performance among the evaluated configurations on BiScopeCap.
Figures & tables
Figure 1: Comparison of BiScopeNet and baseline models, showing how the new data incorporation approach, along with the proposed projection and loss mechanisms, improves hand sub-action recognition, overall action understanding, and object grounding in generated captions.
Figure 2: Overview of BiScopeNet. Video frames are encoded into visual tokens and processed by our FocusAlign, which integrates text-guided cross-attention, adaptive gating, and MLP refinement to prioritise action-relevant patches. The resulting representation is projected into the LLM space to generate fine-grained captions that capture hand interactions underlying the main action.
Dataset
Action
Hand Role
Spatial
Coarse-grained
Fine-grained
Domain
# Train
# Test
Label
Labels
Metadata
Captions
Captions
Source
Samples
Samples
Epic-Kitchens / Visor
✓
✗
✗
✗
✗
Kitchen / Daily Tasks
31,432
3,729
Ego4D
✓
✗
✗
✗
✗
Diverse Activities
22,916
3,304
HOI-QA
✓
✗
✓
✗
✓
Kitchen / Daily Tasks
–
–
HD-Epic
✓
✓
✗
✗
✓
Kitchen / Daily Tasks
38,520
2,846
BiScopeCap (ours)
✓
✓
✓
✓
✓
Diverse Activities
92,868
9,879
Table 1: Comparison between existing egocentric datasets and our proposed BiScopeCap , along with the distribution of samples across its constituent subsets.
Method
# Samples
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
GPT Scoring
200
44.5
31.0
36.0
47.0
Human Scoring
200
42.0
30.3
35.6
51.0
Table 2: Comparison between LLM-as-a-judge scores and human evaluation over 200 predicted captions.
Method
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
BiScopeNet (avg. 3 runs)
45.3
44.7
40.6
58.9
Standard Deviation
± 0.2
± 0.5
± 0.6
± 0.07
Table 3: BiScopeNet stability results across three independent inference and scoring runs on NVILA backbone.
Method
Vision Encoder
Backbone
Trained
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
# Unstruct ↓
SmolVLM [ 33 ]
SigLIP [ 58 ]
-
✗
10.4
11.8
14.6
27.6
4505
Qwen 2.5-VL [ 3 ]
-
-
✗
25.6
29.9
37.0
54.0
1
Qwen 3.5 [ 48 ]
-
-
✗
26.0
32.2
40.6
53.3
6
Intern VL 2.5 [ 7 ]
InternViT [ 8 ]
-
✗
24.8
27.4
37.0
51.5
5
NVILA [ 29 ]
SigLIP
-
✗
14.1
17.2
28.7
39.7
0
NVILA [ 29 ]
SigLIP
-
✓
44.0
43.0
39.1
58.0
0
Table 4: Comparison of BiScopeNet with state-of-the-art VLMs on BiScopeCap. Blue denote the best results.
Table 6: Ablation study of BiScopeNet components using NVILA backbone [ 29 ] on BiScopeCap. The proposed FocusAlign and MetaLoss consistently improve all metrics, demonstrating complementary benefits for sub-action understanding. Blue indicates the best results.
Method
Component
Vision Encoder
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
# Unstruct ↓
VideoChatGPT
Linear + Cross Entropy
CLIP
12.8
10.4
5.9
27.3
374
VideoChatGPT
MLP + Cross Entropy
CLIP
16.8
15.5
12.8
26.4
437
BiScopeNet
FocusAlign + Cross Entropy
CLIP
13.9
10.6
12.9
18.8
196
BiScopeNet
Linear + MetaLoss
CLIP
19.6
19.0
25.8
40.4
0
BiScopeNet
FocusAlign + MetaLoss
CLIP
32.5
18.4
25.8
46.4
5
BiScopeNet
FocusAlign + MetaLoss + CLS
CLIP
30.7
19.4
29.6
47.9
7
Table 7: Ablation study of BiScopeNet components using VideoChatGPT backbone [ 31 ] on the Epic subset of BiScopeCap. The proposed FocusAlign and MetaLoss consistently deliver the best performance, highlighting their complementary benefits for sub-action understanding. Blue indicates the best results.
Method
Trained
Vision Encoder
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
# Unstruct ↓
Ego4D
SmolVLM [ 33 ]
✗
SigLIP
8.9
12.0
16.5
25.8
1256
NVILA [ 29 ]
✗
SigLIP
13.0
16.7
30.9
39.7
0
VideoChatGPT [ 31 ]
✓
CLIP
7.2
10.1
12.3
24.0
563
BiScopeNet (Ours)
✓
DINOv3
22.1
38.7
39.6
42.1
46
Epic+Ego4D
SmolVLM [ 33 ]
✗
SigLIP
13.1
15.6
18.9
29.6
2785
NVILA [ 29 ]
✗
SigLIP
17.7
21.5
34.9
43.1
0
Table 8: Comparison of BiScopeNet using VideoChatGPT [ 31 ] as backbone with state-of-the-art VLMs across Ego4D , Epic+Ego4D , and HD-Epic subsets of BiScopeCap. # Unstruct counts are over the corresponding subset test split, not the full BiScopeCap test set. Blue denote the best results.
Method
Vision Encoder
CLS
% ActRec ↑
% RH ↑
% LH ↑
% ObjRec ↑
# Unstruct ↓
VideoChatGPT
CLIP
✗
12.8
10.4
5.9
27.3
374
BiScopeNet
CLIP
✗
32.5
18.4
25.8
46.4
5
BiScopeNet
CLIP
✓
30.7
19.4
29.6
47.9
7
BiScopeNet
DINOv3
✗
33.1
17.6
36.6
48.0
5
BiScopeNet
DINOv3
✓
28.2
23.0
38.2
49.4
13
BiScopeNet
SigLIP2
✗
32.4
21.6
31.8
47.0
5
Table 9: Ablation study of BiScopeNet integrated into the VideoChatGPT [ 31 ] backbone with different vision encoders (CLIP, DINOv3, and SigLIP2) and the presence or absence of the CLS token on the Epic subset of BiScopeCap. BiScopeNet consistently improves performance across all encoders. SigLIP2+CLS achieves the highest overall and object-level accuracy, while DINOv3 shows superior sensitivity to hand-specific cues, demonstrating the adaptability of BiScopeNet to diverse vision backbones. Blue highlights the best results.
Figure 3: Qualitative comparison of visual understanding across methods.
Figure 4: Qualitative comparison of output structure across methods.
Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.