cs.CVNov 11, 2026

Sub-actions in Action: Text-Guided Hand-Role Alignment for Sub-Action Recognition

Authors: Sadegh Rahmaniboldaji, Filip Rybansky, Quoc Vuong, Frank Guerin, Andrew Gilbert

Organizations: University of Surrey, Surrey, UK · University of Newcastle, Newcastle, UK

Abstract

Our objective is to enhance VLMs ability to distinguish the roles of the left and right hands in egocentric actions. We introduce sub-action detection, a task that requires models to explicitly recognise the left- and right-hand roles and their manipulated objects within an ongoing high-level action. To address this challenge, we propose BiScopeNet, a framework that augments existing video-LLMs without modifying their vision encoders or language models. BiScopeNet incorporates two components: FocusAlign, a gated text-guided attention-based module that selectively aligns video features with textual context to emphasise action-relevant cues, and MetaLoss, a masked, object-weighted loss that prioritises visually grounded terms. To supervise sub-action detection, we introduce BiScopeCap-100K, a large-scale video-level dataset curated from existing egocentric datasets, providing dense captions that jointly describe high-level actions, left/right-hand roles, and manipulated objects, enriched with spatial metadata for grounding. Across the evaluated Video-LLM backbones, BiScopeNet improves action, hand-role, and object recognition over the corresponding supervised baselines, achieving the strongest aggregate performance among the evaluated configurations on BiScopeCap.

Figures & tables

Explore similar work

CardsList
  1. Beyond Scene Shortcuts: Action-Centric Video-Text Alignment for Fine-Grained Action Understanding

    Nov 4, 2026Zhiliang Zhu, Jiazheng Huang, Dawei Zhaofine-grained action understandingVideo-Language Models

  2. Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition

    Jun 16, 2026Alessandro Sottovia, Alessandro Torcinovich, Oswald LanzZero-Shot LearningMulti-Agent Debate

  3. From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models

    Jun 13, 2026Bessie Dominguez-Dager, Francisco Gomez-Donoso, Miguel Cazorla +3Egocentric Video UnderstandingIn-Context Learning