Vision-language models widely adopt Multimodal Rotary Position Embedding (MRoPE), an extension of Rotary Position Embedding (RoPE) that incorporates temporal, height, and width axes to encode the spatial structure of visual tokens. However, MRoPE also assigns height and width coordinates to text tokens that lack inherent spatial structure, introducing spatial positional effects into text-vision attention. Existing MRoPE variants explore different frequency allocations and coordinate layouts, with their designs suggesting different intuitions about these cross-modal effects: they may provide useful spatial references for associating language with visual regions, or introduce unnecessary positional preferences that hinder text-vision interactions. In this work, we disentangle intra-visual spatial structure from cross-modal spatial effects to investigate this question. We identify a constraint of shared coordinate layouts that prevents eliminating cross-modal spatial effects while preserving intra-visual spatial encoding, and develop controlled experiments to isolate their respective contributions. Our results across multiple model scales, MRoPE variants, and benchmarks show that cross-modal spatial effects provide little additional benefit as spatial references and can even impair model performance, while intra-visual spatial encoding remains useful for fine-grained grounding.
Figures & tables
Figure 1: Illustration of different coordinate layouts for vanilla-RoPE and representative MRoPE variants in the temporal-height-width ( t,h,w ) space. Blue dots ( ∙ ) represent text tokens, orange squares ( ■ ) represent visual patches within an image, and the red star ( ★ ) marks a query text token x . The orange color gradient indicates the relative spatial proximity of visual patches to x along the height and width axes, with lighter colors indicating nearer positions and darker colors indicating farther positions.
Figure 2: Overview of attention-specific coordinate assignments for disentangling cross-modal and intra-visual spatial effects. (a) Two coordinate assignments for visual patches. The original coordinates pa=(timg,ha,wa) preserve each patch’s spatial location, while the neutral coordinates p~a=(timg,h~,w~) assign identical height and width coordinates to all patches within an image. Here, h~=w~=timg for the diagonal text layout. (b) Attention-specific coordinate assignments. Independently selecting original or neutral coordinates for text-vision (T-V) and vision-vision (V-V) attention yields four controlled configurations, while text coordinates and text-text (T-T) attention remain unchanged. (c) Geometric view on the (h,w) plane. In the w/o T-V Spatial configuration, original coordinates preserve spatial relationships among visual patches in V-V attention, while neutral coordinates yield identical relative rotations between any text token and all visual patches in T-V attention. This enables independent evaluation of the two spatial effects.
Configuration
Notation
T-V Attention
V-V Attention
Original
Original
pa
pa
Cross-Modal Neutral
w/o T-V Spatial
pa
pa
Intra-Visual Neutral
w/o V-V Spatial
pa
pa
Fully Neutral
w/o Both Spatial
pa
pa
Table 1: Controlled configurations based on attention-specific visual patch coordinates. T-V and V-V denote text-vision and vision-vision attention, respectively. Each entry specifies the visual patch coordinates used for the corresponding attention, while text token coordinates remain unchanged. pa and pa denote the original and spatially neutral coordinates for visual patch tokens.
Table 2: Evaluation of official Qwen3-VL models under different attention-specific coordinate configurations across three model scales. General evaluates visual understanding and spatial understanding, where Visual and Spatial report the average scores across 10 visual understanding and 4 spatial understanding benchmarks, respectively. Grounding evaluates referring expression grounding, reporting accuracy at IoU thresholds of 0.5, 0.7, and 0.9, averaged across RefCOCO, RefCOCO+, and RefCOCOg. Higher scores indicate better performance.
Benchmark
Qwen3-VL-2B
Qwen3-VL-4B
Original
w/o T-V
w/o V-V
w/o Both
Original
w/o T-V
w/o V-V
w/o Both
Spatial
Spatial
Spatial
Spatial
Spatial
Spatial
Visual
MMStar
44.5 ± 0.1
44.6 ± 0.0
44.2 ± 0.3
45.2 ± 0.7
48.8 ± 0.4
50.0 ± 1.1
48.8 ± 0.8
49.5 ± 0.1
SEEDBench
70.9 ± 0.3
71.2 ± 0.1
71.1 ± 0.2
70.8 ± 0.1
72.5 ± 0.3
72.8 ± 0.3
72.9 ± 0.0
73.0 ± 0.1
MMBench
68.9 ± 0.4
69.4 ± 0.3
69.2 ± 0.7
68.1 ± 0.3
72.9 ± 0.9
73.7 ± 1.2
73.4 ± 0.3
73.6 ± 0.2
RealWorldQA
62.5 ± 0.8
63.0 ± 1.0
60.0 ± 1.2
60.7 ± 1.7
64.8 ± 0.9
65.9 ± 0.6
63.3 ± 0.8
65.7 ± 0.3
Table 3: Controlled training results of Qwen3-VL-2B/4B under our four spatial configurations using MRoPE-I [ 4 ] . All configurations follow the same training protocol. Results are reported as mean ± std over three random seeds (42, 123, 3407). Visual and Spatial denote general visual understanding and spatial understanding benchmarks, respectively. The final rows report average scores across the 10 visual and 4 spatial benchmarks. Higher values indicate better performance.
Figure 3: Controlled training results across two MRoPE variants (VideoRoPE [ 25 ] , HoPE [ 8 ] ) under four spatial configurations (seed 42). Each row represents an MRoPE variant, while columns report average visual understanding, spatial understanding, and referring expression grounding performance. Blue circles and orange squares denote Qwen3-VL-2B and Qwen3-VL-4B, respectively, with their corresponding scores shown on the left and right y-axes. The shaded regions indicate configurations without intra-visual spatial encoding. More quantitative results on IL-RoPE [ 9 ] can be found in Appendix 6 .
Figure 4: Visualization of attention maps for Qwen3-VL-4B [ 20 ] (middle layers, 12-23) using the MRoPE-I [ 4 ] under four spatial configurations. The orange-highlighted words indicate the query text token used to compute attention over visual patches. Warmer colors indicate higher relative attention intensity. More visualizations can be found in Appendix 10 .
Table 4: Controlled training results on referring expression grounding using Qwen3-VL-2B/4B with MRoPE-I. IoU reports mean intersection-over-union, while [email protected], [email protected], and [email protected] measure localization accuracy at the corresponding IoU thresholds. All metrics are aggregated over the evaluated splits of RefCOCO, RefCOCO+, and RefCOCOg and reported as percentages (mean ± std over three random seeds: 42, 123, 3407). Higher values indicate better performance.
Table 5: Controlled training results on Qwen3-VL-4B with ILRoPE [ 9 ] (seed 42). Visual and Spatial report the average scores across 10 visual understanding and 4 spatial understanding benchmarks, respectively. IoU reports mean intersection-over-union, while [email protected], [email protected], and [email protected] measure localization accuracy at the corresponding IoU thresholds. All metrics are aggregated over the evaluated splits of RefCOCO, RefCOCO+, and RefCOCOg and reported as percentages. Higher values indicate better performance.
Configuration
Stage 1
Stage 2
Trainable Modules
Projector + Merger
All Modules
Max Length
8,192
8,192
Image Resolution
Dynamic
Dynamic
Image Pixel Range
1,024–1,048,576
1,024–1,048,576
Attention Backend
Eager
Eager
Optimizer
AdamW
AdamW
Table 6: Hyperparameter configurations for the two-stage controlled training protocol. Batch size denotes the effective global batch size.
Figure 5: Visualization of attention maps for Qwen3-VL-4B using the MRoPE-I under four spatial configurations. The orange-highlighted word cat indicates the query text token used to compute attention over visual patches. Columns correspond to Original, w/o T-V Spatial, w/o V-V Spatial, and w/o Both Spatial. Rows represent attention averaged over shallow (layers 0-11), middle (layers 12-23), and deep (layers 24-35) layers. Warmer colors indicate higher relative attention intensity.
Figure 6: Visualization of attention maps for Qwen3-VL-4B using the VideoRoPE under four spatial configurations. The orange-highlighted word cat indicates the query text token used to compute attention over visual patches. Columns correspond to Original, w/o T-V Spatial, w/o V-V Spatial, and w/o Both Spatial. Rows represent attention averaged over shallow (layers 0-11), middle (layers 12-23), and deep (layers 24-35) layers. Warmer colors indicate higher relative attention intensity.
Figure 7: Visualization of attention maps for Qwen3-VL-4B using the HoPE under four spatial configurations. The orange-highlighted word cat indicates the query text token used to compute attention over visual patches. Columns correspond to Original, w/o T-V Spatial, w/o V-V Spatial, and w/o Both Spatial. Rows represent attention averaged over shallow (layers 0-11), middle (layers 12-23), and deep (layers 24-35) layers. Warmer colors indicate higher relative attention intensity.
Figure 8: Visualization of attention maps for Qwen3-VL-4B using the MRoPE-I under four spatial configurations. The orange-highlighted word green indicates the query text token used to compute attention over visual patches. Columns correspond to Original, w/o T-V Spatial, w/o V-V Spatial, and w/o Both Spatial. Rows represent attention averaged over shallow (layers 0-11), middle (layers 12-23), and deep (layers 24-35) layers. Warmer colors indicate higher relative attention intensity.
Figure 9: Visualization of attention maps for Qwen3-VL-4B using the VideoRoPE under four spatial configurations. The orange-highlighted word green indicates the query text token used to compute attention over visual patches. Columns correspond to Original, w/o T-V Spatial, w/o V-V Spatial, and w/o Both Spatial. Rows represent attention averaged over shallow (layers 0-11), middle (layers 12-23), and deep (layers 24-35) layers. Warmer colors indicate higher relative attention intensity.
Figure 10: Visualization of attention maps for Qwen3-VL-4B using the HoPE under four spatial configurations. The orange-highlighted word green indicates the query text token used to compute attention over visual patches. Columns correspond to Original, w/o T-V Spatial, w/o V-V Spatial, and w/o Both Spatial. Rows represent attention averaged over shallow (layers 0-11), middle (layers 12-23), and deep (layers 24-35) layers. Warmer colors indicate higher relative attention intensity.
Despite the remarkable capabilities of Multimodal Large Language Models (MLLMs), they still suffer from visual fading in long-context scenarios. Specifically, the attention to visual tokens diminishes as the text sequence lengthens, leading to text generation detached from visual constraints. We attribute this degradation to the inherent inductive bias of Multimodal RoPE, which penalizes inter-modal attention as the distance between visual and text tokens increases. To address this, we propose inter-modal Distance Invariant Position Encoding (DIPE), a simple but effective mechanism that disentangles position encoding based on modality interactions. DIPE retains the natural relative positioning for intra-modal interactions to preserve local structure, while enforcing an anchored perceptual proximity for inter-modal interactions. This strategy effectively mitigates the inter-modal distance-based penalty, ensuring that visual signals remain perceptually consistent regardless of the context length. Experimental results demonstrate that by integrating DIPE with Multimodal RoPE, the model maintains stable visual grounding in long-context scenarios, significantly alleviating visual fading while preserving performance on standard short-context benchmarks. Code is available at https://github.com/lchen1019/DIPE.
Lin Chen, Bolin Ni, Qi Yang +5
1MAIS, Institute of Automation, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · 3Tencent +1
Vision-Language Models (VLMs) achieve strong cross-modal performance, yet recent evidence suggests they over-rely on textual descriptions while under-utilizing visual evidence -- a phenomenon termed ``text shortcut learning.'' We propose an adversarial evaluation framework that quantifies this cross-modal dependency by measuring accuracy degradation (Drop) when semantically conflicting text is paired with unchanged images. Four adversarial strategies -- shape_swap, color_swap, position_swap, and random_text -- are applied to a controlled geometric-shapes dataset (n=1,000). We compare three configurations: Baseline CLIP (ViT-B/32), LoRA fine-tuning, and LoRA Optimized (integrating Hard Negative Mining, Label Smoothing, layer-wise learning rates, Cosine Restarts, curriculum learning, and data augmentation). The optimized model reduces average Drop from 27.5% to 9.8% (64.4% relative improvement, p<0.001) while maintaining 97% normal accuracy. Attention visualization and embedding-space analysis confirm that the optimized model attends more to visual features and achieves tighter cross-modal alignment.
Lijie Zhou
School of Computer Science University of Nottingham Ningbo China Ningbo, China
Multimodal Reasoning Models (MRMs) leveraging Chain-of-Thought (CoT) based thinking have revolutionized mathematical and logical problem-solving. However, we show that this paradigm struggles with generalized spatial intelligence. We perform a comprehensive evaluation of seventeen models across thirteen spatial benchmarks and identify a critical gap: CoT prompting consistently degrades performance in visual spatial reasoning. Furthermore, through a novel No-Image++ ablation, we demonstrate that MRMs and CoT prompted MLMs suffer from severe shortcut learning, and hallucinate visual details from textual priors even when the image is absent. These findings challenge the efficacy of text-only CoT for spatial tasks and underscore the need for vision-centric reasoning paradigms.
Sai Srinivas Kancheti, Aditya Sanjiv Kanade, Vineeth N. Balasubramanian +1