cs.CVNov 4, 2026

Beyond Scene Shortcuts: Action-Centric Video-Text Alignment for Fine-Grained Action Understanding

Authors: Zhiliang Zhu, Jiazheng Huang, Dawei Zhao

Organizations: Harbin Engineering University, China · South China Normal University, China

Abstract

Fine-grained action understanding requires vision-language models to ground what an agent does, how it acts, and why it acts in spatio-temporal evidence. However, richer supervision can backfire: fine-tuned models may generate detailed action descriptions grounded in static scenes and salient objects rather than motion, a failure we call Environment-Grounded Action Hallucination (EGAH). To mitigate EGAH, we propose Staged Attention Rectification and Reinforcement (STARR), an action-centric video-text alignment framework that breaks scene-action shortcuts through Action-Environment Dissociation and reinforces action-focused grounding with an Attention Consistency Reward within Group Relative Policy Optimization. We further construct FineActionR&R, combining temporally localized captions with hierarchical and temporal-causal reasoning data. Built on Qwen2.5-VL-7B, FineActionVLM-R1 improves average MotionBench accuracy by 3.01 percentage points over its base model and achieves 91.50% on HAU-MCQ-Bench. Qualitative comparisons further indicate more faithful fine-grained action descriptions with fewer environment-induced hallucinations.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. InstrAct: Towards Action-Centric Understanding in Instructional Videos

    Apr 9, 2026Zhuoyi Yang, Jiapeng Yu, Reuben Tan +2audio-language pretrainingVideo Understanding

  2. FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding

    May 19, 2026Gueter Josmy Faure, Min-Hung Chen, Jia-Fong Yeh +2VLM EvaluationTemporal Video Grounding

  3. From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models

    Jun 13, 2026Bessie Dominguez-Dager, Francisco Gomez-Donoso, Miguel Cazorla +3Egocentric Video UnderstandingIn-Context Learning