Beyond Scene Shortcuts: Action-Centric Video-Text Alignment for Fine-Grained Action Understanding
Organizations: Harbin Engineering University, China · South China Normal University, China
Abstract
Fine-grained action understanding requires vision-language models to ground what an agent does, how it acts, and why it acts in spatio-temporal evidence. However, richer supervision can backfire: fine-tuned models may generate detailed action descriptions grounded in static scenes and salient objects rather than motion, a failure we call Environment-Grounded Action Hallucination (EGAH). To mitigate EGAH, we propose Staged Attention Rectification and Reinforcement (STARR), an action-centric video-text alignment framework that breaks scene-action shortcuts through Action-Environment Dissociation and reinforces action-focused grounding with an Attention Consistency Reward within Group Relative Policy Optimization. We further construct FineActionR&R, combining temporally localized captions with hierarchical and temporal-causal reasoning data. Built on Qwen2.5-VL-7B, FineActionVLM-R1 improves average MotionBench accuracy by 3.01 percentage points over its base model and achieves 91.50% on HAU-MCQ-Bench. Qualitative comparisons further indicate more faithful fine-grained action descriptions with fewer environment-induced hallucinations.
Figures & tables
| # | Model | Organization | Frames | LLM Params | Dev Avg (%) | MR (%) | LM (%) | CM (%) | MO (%) | AO (%) | RC (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Qwen3-VL-30B-A3B-Instruct | Alibaba | 2fps | 30B (A3B) | 57.59 | 61.03 | 61.90 | 55.84 | 75.07 | 45.09 | 26.75 |
| 2 | Qwen3-VL-8B-Instruct | Alibaba | 2fps | 8B | 55.55 | 59.07 | 57.88 | 55.84 | 72.46 | 44.70 | 24.00 |
| 3 | Kimi-VL-2506 | Kimi | 2fps | 16B(A3B) | 54.6 | 58.7 | 55.78 | 45.64 | 73.48 | 42.08 | 22.52 |
| 4 | Qwen2.5-VL-72B-Instruct | Alibaba | 1fps | 72B | 57.00 | 58.00 | 61.00 | 63.00 | 72.00 | 47.00 | 31.00 |
| 5 | gemini-2.5-flash | 2fps | - | 56.12 | 59.25 | 59.97 | 59.9 | 72.55 | 44.4 | 29.14 | |
| 6 | GLM-4.5V | Zhipu AI | 2fps | 106B(A12B) | 61.42 | 65.08 | 65.16 | 58.72 | 77.38 | 48.34 | 40.37 |
| Model | Avg. Accuracy(%) | Seen Videos + Unseen Questions.(%) | Unseen Videos + Unseen Questions.(%) |
|---|---|---|---|
| Qwen3-VL-8B-Instruct | 63.75 | 60.95 | 66.95 |
| Qwen3-VL-8B-Thinking | 67.2 | 65.92 | 68.66 |
| Qwen3-VL-32B-Instruct | 78.75 | 76.12 | 81.77 |
| Qwen3-VL-32B-Thinking | 75.43 | 72.64 | 78.63 |
| Qwen3-VL-30B-A3B-Instruct | 72.11 | 71.39 | 72.93 |
| Qwen3-VL-30b-A3B-Thinking | 71.31 | 72.93 | 69.9 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Training | MB | AH | Recall |
|---|---|---|---|
| Base | 54.83 | 29.78 | 66.41 |
| Matched SFT | 55.27 | 28.76 | 70.23 |
| SFT+AED | 55.83 | 25.24 | 71.91 |
| SFT+G | 55.69 | 26.60 | 71.53 |
| SFT+AED+G | 56.21 | 23.53 | 72.74 |
| SFT+AED+G+A | 56.68 | 21.30 | 74.28 |
| Comparison (treatment control) | AH | 95% CI |
|---|---|---|
| SFT+AED matched SFT | ||
| Add ACR to SFT+AED+G, without ES | ||
| R1 Base at 4 FPS |
| MotionBench | AH | |||
|---|---|---|---|---|
| FPS | Base | R1 | Base | R1 |
| 1 | 53.43 | 56.29 | 32.02 | 22.16 |
| 2 | 54.83 | 57.84 | 29.78 | 19.71 |
| 4 | 55.82 | 59.13 | 28.71 | 17.60 |
| Model/input | Overall | Unseen |
|---|---|---|
| Base | 45.29 | 45.30 |
| Matched SFT | 85.86 | 82.83 |
| R1 | 91.50 | 90.31 |
| R1, no video | 37.05 | 33.36 |
| Selection | Temporal | Spatiotemporal |
|---|---|---|
| Matched random | 39.71 | 20.46 |
| Reference top 25% | 64.54 | 44.06 |
| Training | Correct pairs/100 | AH (%) |
|---|---|---|
| Matched SFT | 54 | 28.76 |
| SFT+AED | 71 | 25.24 |
| Model (7B) | Video-MME accuracy (%) |
|---|---|
| Base | 56.00 |
| Matched SFT | 56.45 |
| Full | 57.77 |
| Qwen2.5-VL-3B | MotionBench | AH |
|---|---|---|
| Base | 51.49 | 31.22 |
| Full training | 53.40 | 23.89 |