cs.CVOct 30, 2026

GRIT: Gradient Masking for Visually Misgrounded Tokens in Text-Rich Reasoning

Authors: Chanseul Cho, Noori Bae, Dongheon Lee, Youngmin Ro

Organizations: Machine Intelligence Laboratory, University of Seoul Data Science Interdisciplinary Major

Abstract

Reinforcement learning with verifiable rewards (RLVR) has recently been extended to vision-language models (VLMs), yet its effectiveness remains limited for text-rich reasoning tasks. We highlight a key factor contributing to this limitation: models frequently arrive at correct final answers despite misperceiving visual information during intermediate steps. We define this phenomenon as Correct-but-Visually-Misgrounded Reasoning (CVMR). Because standard RLVR rewards only final-answer correctness, it fails to differentiate between properly grounded and misgrounded reasoning, reinforcing both equally. Through token-level analysis, we reveal that CVMR tokens exhibit distinct attention patterns that differ systematically from well-grounded ones. Based on this observation, we propose GRIT, a training framework that incorporates visual grounding into RLVR to mitigate the impact of CVMR. Furthermore, to address the lack of text-rich reasoning benchmarks that allow for systematic evaluation across text length and reasoning depth, we introduce GRIT-Bench. Experiments demonstrate that GRIT significantly improves both reasoning accuracy and grounding fidelity over existing RLVR methods.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models

    May 2, 2026Yin Zhang, Jiaxuan Zhao, Zonghan Wu +5VLM ReasoningVision-Language Models

  2. iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning

    May 29, 2026Chang-Bin Zhang, Yujie Zhong, Qiang Zhang +1Large Vision-Language ModelsVision-Language Grounding