Multimodal Large Language Models (MLLMs) struggle with tasks that require reasoning about 2D object orientation in images, as documented in prior work. Tong et al. and Nichols et al. hypothesize that these failures originate in the visual encoder, since commonly used encoders such as CLIP and SigLIP are trained for image-text semantic alignment rather than geometric reasoning. We design a controlled empirical protocol to test this claim by measuring whether rotations can be recovered from encoder representations. In particular, we examine SigLIP and ViT features from LLaVA OneVision and Qwen2.5-VL-7B-Instruct models, respectively, using full images, and examine CLIP representations in LLaVA 1.5 and 1.6 using rotated foreground patches against natural background images. Our null hypothesis is that orientation information is not preserved in the encoder embeddings and we test this by training linear regressors to predict object orientation from encoded features. Contrary to the hypothesis, we find that orientation information is recoverable from encoder representations: simple linear models accurately predict object orientations from embeddings. This contradicts the assumption that MLLM orientation failures originate in the visual encoder. Having rejected the accepted hypothesis that MLLMs struggle with 2D orientation tasks because of visual encoder limitations, we still don't know why they fail. Although a full explanation is beyond the scope of this paper, we show that although present, orientation information is spread diffusely across tens of thousands of features. This may or may not be while MLLMs fail to exploit the available orientation information.
Figures & tables
Image Set
Prompt
Whole Images
How much is the 2nd image rotated clockwise when compared to the first image?
In-Place Rotated Images
How much is the < object(s) > rotated clockwise in degrees when compared to the first image?
dog on beach
By how much is the dog in the center inside the circle rotated if it is known that the rotation is zero degrees when the dog’s legs are vertical?
lizard on fish
By how much is the lizard in the center inside the circle rotated if it is known that the rotation is zero degrees when the lizard is roughly horizontal with its tail pointing left and its head pointing right?
train on indoor
By how much is the train in the center inside the circle rotated if it is known that the rotation is zero degrees when the train is vertical with the train tracks at the bottom and the steam from the train going vertically upwards?
Table 1: Image sets and corresponding prompts to LLaVA/Qwen2.5-VL-7B-Instruct.
Figure 1: Set of images - Sets A and B are used for experiments with LLaVA-OV and Qwen2.5-VL-7B-Instruct, and set C is used for LLaVA1.5 and LLaVA1.6. Since LLaVA-OV and Qwen2.5-VL-7B-Instruct can be prompted with 2 images, real images were used. But since LLaVA1.5 and 1.6 can be prompted with only 1 image, synthetic images with reduced artifacts were used to improve the prediction accuracy. [A: Whole Images] : (a) dog (b) train (c) lizard (d) beach (e) indoor (f) fish. [B: In-Place Rotated Objects] : (a) koala on beach (b) vase on indoor (c) vase and toaster on indoor. [C: Blended Images with foreground on background] : (a) dog on beach (b) lizard on fish (c) train on indoor environment, with foreground scales 1, 2 and 3 from left to right.
Figure 2: Collage of every 15th image from Sections A (a) dog (whole image) and B (a) koala (in-place rotated) of Figure 1 .
Figure 3: 2D orientation estimation performance comparison of LLaVA-OneVision and Qwen2.5-Vl-7B-Instruct on the images with the dog for 36 randomly selected images.
Two Image Version Whole Images LLaVA-OV Qwen Instruct MAE Max Min MAE Max Min Dog 0.67 1.61 0.03 1.30 4.63 0.09 Lizard 0.44 1.63 0.01 2.20 6.31 0.06 Train 0.56 2.18 .01 2.43 10.01 0.29 Beach 0.74 2.67 0.03 1.63 4.98 0.03 Indoor 0.81 2.92 0.003 1.32 4.96 0.01 Fish 2.31 5.31 0.12 2.85 8.62 0.17 In-Place Rotated Object(s) Koala 0.36 1.47 0.01 1.01 3.66 0.06 Vase 0.48 1.76 0.003 0.90 4.18 0.01 Vase & Toaster 0.65 1.88 0.02 1.58 5.81 0.02
Single Image Version Blended Images LLaVA 1.5 LLaVA 1.6 MAE Max Min MAE Max Min Dog Scale 1 1.41 4.19 0.06 0.62 2.40 0.001 Scale 2 0.89 3.20 0.0001 0.6 2.01 0.01 Scale 3 0.67 2.52 0.02 0.52 2.2 0.03 Train Scale 1 1.3 3.79 0.02 0.78 2.12 0.02 Scale 2 0.9 2.69 0.003 0.53 1.72 0.0002 Scale 3 0.72 2.12 0.008 0.42 1.2 0.005 Lizard Scale 1 2.08 7.24 0.0003 0.87 3.15 0.04 Scale 2 1.25 4.67 0.005 0.64 2.42 0.02 Scale 3 1.13 3.53 0.04 0.71 1.69 0.05
Table 2: LLaVA-OneVision , Qwen2.5-VL-7B-Instruct , LLaVA 1.5 and LLaVA 1.6 vision encoders estimate 2D orientation to a high degree of accuracy across diverse image sets.
Table 3: Low Kolmogorov–Smirnov (K-S) statistic coupled with p -value > alpha (0.05) means there’s no reason to reject the null hypothesis that the error distribution for LLaVA-OneVision , Qwen2.5-VL-7B-Instruct , LLaVA 1.5 and 1.6 are purely random (Gaussian).
Figure 4: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with dog. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian.
Figure 5: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with dog. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian.
LLaVA OneVision
Qwen2.5-VL-7B-Instruct
Angle ( ∘ )
Count
Angle ( ∘ )
Count
90
149
90
142
180
28
10-15
14
0
2
10
11
45
1
180
6
Table 4: Top LLaVA-OneVision and Qwen2.5-VL-7B-Instruct query responses for the 180 samples with the dog scene. Both models frequently respond that the 2D orientation is 90 ∘ .
Figure 6: Incremental feature substitution for LLaVA-OneVision on images with the dog scene. On the y axis, when y = 1, predicted value matches the target orientation and when y = 0, predicted value matches the anchor orientation. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 7: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the dog scene. On the y axis, when y = 1, predicted value matches the target orientation and when y = 0, predicted value matches the anchor orientation. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 8: Collage of every 20th image from the images with the dog foreground (biggest foreground)
Figure 9: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with dog for 36 randomly selected images
Figure 10: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with lizard for 36 randomly selected images
Figure 11: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with fish for 36 randomly selected images
Figure 12: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with train for 36 randomly selected images
Figure 13: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with indoor for 36 randomly selected images
Figure 14: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with koala for 36 randomly selected images
Figure 15: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with vase for 36 randomly selected images
Figure 16: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with vase and toaster for 36 randomly selected images
Figure 17: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with dog foregrounds (scale 1) for 72 randomly selected images
Figure 18: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with dog foregrounds (scale 2) for 72 randomly selected images
Figure 19: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with dog foregrounds (scale 3) for 72 randomly selected images
Figure 20: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with lizard foregrounds (scale 1) for 72 randomly selected images
Figure 21: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with lizard foregrounds (scale 2) for 72 randomly selected images
Figure 22: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with lizard foregrounds (scale 3) for 72 randomly selected images
Figure 23: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with train foregrounds (scale 1) for 72 randomly selected images
Figure 24: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with train foregrounds (scale 2) for 72 randomly selected images
Figure 25: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with train foregrounds (scale 3) for 72 randomly selected images
Figure 26: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with lizard. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 27: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with lizard. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 28: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with train. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 29: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with train. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 30: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with beach. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 31: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with beach. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 32: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 33: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 34: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with fish. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 35: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with fish. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 36: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with koala-beach. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 37: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with koala-beach. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 38: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with vase-indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 39: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with vase-indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 40: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with vase-toaster-indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 41: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with vase-toaster-indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 42: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with dog foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 43: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with dog foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 44: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with dog foregrounds (Scale 1). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 45: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with dog foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 46: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with dog foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 47: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with lizard foregrounds (Scale 1). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 48: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with lizard foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 49: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with lizard foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 50: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with lizard foregrounds (Scale 1). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 51: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with lizard foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 52: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with lizard foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 53: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with train foregrounds (Scale 1). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 54: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with train foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 55: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with train foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 56: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with train foregrounds (Scale 1). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 57: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with train foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 58: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with train foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
LLaVA 1.5
LLaVA 1.6
Angle ( ∘ )
Count
Angle ( ∘ )
Count
90
25
90
40
not possible
10
not possible
17
45
9
0
6
0
8
no answer
4
no answer
4
180
2
Table 5: LLaVA-LLaMA results for the 72 test samples with the beach scene in the background. LLaMA frequently responds that the 2D orientation is 90 ∘ or “not possible to determine”. The number of correct answers (among valid responses) is very low, even with high thresholds
Figure 59: Incremental feature substitution for LLaVA-OneVision on images with the lizard scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 60: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the lizard scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 61: Incremental feature substitution for LLaVA-OneVision on images with the train scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 62: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the train scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 63: Incremental feature substitution for LLaVA-OneVision on images with the beach scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 64: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the beach scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 65: Incremental feature substitution for LLaVA-OneVision on images with the indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 66: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 67: Incremental feature substitution for LLaVA-OneVision on images with the fish scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 68: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the fish scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 69: Incremental feature substitution for LLaVA-OneVision on images with the koala-beach scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 70: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the koala-beach scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 71: Incremental feature substitution for LLaVA-OneVision on images with the vase-indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 72: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the vase-indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 73: Incremental feature substitution for LLaVA-OneVision on images with the vase-toaster-indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 74: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the vase-toaster-indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 75: Incremental feature substitution for LLaVA 1.5 on images with the beach background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 20,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 76: Incremental feature substitution for LLaVA 1.6 on images with the beach background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 16,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 77: Incremental feature substitution for LLaVA 1.5 on images with the fish background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 20,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 78: Incremental feature substitution for LLaVA 1.6 on images with the fish background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 16,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 79: Incremental feature substitution for LLaVA 1.5 on images with the indoor background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 20,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 105 .) This implies the orientation information is highly diffuse.
Figure 80: Incremental feature substitution for LLaVA 1.6 on images with the indoor background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 16,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times 106 .) This implies the orientation information is highly diffuse.
Figure 81: Results of foreground orientation estimation by LLaVA 1.5 for dog images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
Figure 82: Images with synthetic backgrounds used to test the impact of background rotations on foreground orientation estimation.
Figure 83: Results of foreground orientation estimation by LLaVA 1.6 for dog images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
Figure 84: Results of foreground orientation estimation by LLaVA 1.5 for lizard images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
Figure 85: Results of foreground orientation estimation by LLaVA 1.6 for lizard images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
Figure 86: Results of foreground orientation estimation by LLaVA 1.5 for train images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
Figure 87: Results of foreground orientation estimation by LLaVA 1.6 for train images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
S/N
Train Set
Test set
MAE (degrees)
LLaVA 1.5
LLaVA 1.6
1
dog on chessboard (FG rotated)
baseline (FG rotated)
1.1
0.71
BG + FG rotated
96.5
92.34
BG rotated
80.09
80.48
dog on grid (FG rotated)
baseline (FG rotated)
1.28
0.68
BG + FG rotated
92.13
88.14
Table 6: Impact of background (BG) image rotation on foreground (FG) rotation (rot.) estimation for dog foreground images (Scale 1) using LLaVA 1.5 and LLaVA 1.6 - Mean Absolute Error (MAE) for synthetic background image sets under different rotation conditions. When background is rotated, performance: (1) degrades sharply when trained on only FG rot. images, (2) and (3) improves significantly when trained on BG+FG rot., (4) improves moderately when trained on only BG rot.
S/N
Train Set
Test set
MAE (degrees)
LLaVA 1.5
LLaVA 1.6
1
lizard on chessboard (FG rotated)
baseline (FG rotated)
1.81
1.36
BG + FG rotated
85.2
85.55
BG rotated
80.2
79.93
lizard on grid (FG rotated)
baseline (FG rotated)
1.87
1.16
BG + FG rotated
84.1
87.22
Table 7: Impact of background (BG) image rotation on foreground (FG) rotation (rot.) estimation for lizard foreground images (Scale 1) using LLaVA 1.5 and LLaVA 1.6 - Mean Absolute Error (MAE) for synthetic background image sets under different rotation conditions. When background is rotated, performance: (1) degrades sharply when trained on only FG rot. images, (2) and (3) improves significantly when trained on BG+FG rot., (4) improves moderately when trained on only BG rot.
S/N
Train Set
Test set
MAE (degrees)
LLaVA 1.5
LLaVA 1.6
1
train on chessboard (FG rotated)
baseline (FG rotated)
1.2
0.77
BG + FG rotated
86.59
82.96
BG rotated
80.46
81
train on grid (FG rotated)
baseline (FG rotated)
1.3
0.94
BG + FG rotated
87.4
87.96
Table 8: Impact of background (BG) image rotation on foreground (FG) rotation (rot.) estimation for train foreground images (Scale 1) using LLaVA 1.5 and LLaVA 1.6 - Mean Absolute Error (MAE) for synthetic background image sets under different rotation conditions. When background is rotated, performance: (1) degrades sharply when trained on only FG rot. images, (2) and (3) improves significantly when trained on BG+FG rot., (4) improves moderately when trained on only BG rot.
Humans learn object orientation progressively, from recognizing which way an object faces, to mentally rotating it, to reasoning about orientations between objects. Current vision-language benchmarks largely conflate orientation with position and general scene understanding. We introduce Discriminative Orientation Reasoning Intelligence (DORI), a cognitively grounded hierarchical benchmark that makes object orientation the primary target. Inspired by stages of human orientation cognition, DORI decomposes orientation into four dimensions, each evaluated at coarse (categorical) and granular (metric) levels. Composed from 13,652 images across 14 sources, DORI provides 33,656 multiple-choice questions covering 67 object categories in real-world and synthetic settings. Its coarse-to-granular design isolates orientation from confounds such as object recognition difficulty, scene clutter, and linguistic ambiguity via bounding-box isolation, standardized spatial reference frames, and structured prompts. Evaluating 24 state-of-the-art vision-language models shows a clear pattern: models that perform well on general spatial benchmarks are near-random on object-centric orientation tasks. The best models reach only 54.2% on coarse and 45.0% on granular judgments, with largest failures on compound rotations and shifts in inter-object reference frames. Large coarse-to-granular gaps reveal reliance on categorical heuristics rather than geometric reasoning, a limitation hidden by existing benchmarks. These results identify orientation understanding as an unsolved challenge for multimodal systems, with implications for robotic manipulation, 3D scene reconstruction, and human-AI interaction.
Nazia Tasnim, Keanu Nichols, Yuting Yang +4
Boston University · Johns Hopkins University · Runway
Humans develop object orientation understanding progressively, from recognizing which way an object faces to reasoning about orientations across multiple objects. Yet existing vision-language benchmarks largely conflate orientation with broader spatial reasoning. We introduce Discriminative Orientation Reasoning Intelligence (DORI), a cognition-informed hierarchical benchmark that establishes object orientation as the primary evaluation target. DORI decomposes orientation into four dimensions, each evaluated at coarse (categorical) and granular (metric) levels, yielding 33,656 multiple-choice questions over 13,652 real-world and synthetic images from 14 sources. Its design isolates orientation from confounding factors through object isolation, standardized reference frames, and structured prompts. Evaluating 26 state-of-the-art vision-language models reveals a consistent limitation: models strong on general spatial benchmarks remain near-random on object-centric orientation reasoning. The best model achieves only 64.2% on coarse and 42.9% on granular judgments, with the largest drops on compound rotations and inter-object reference frame shifts. Large coarse-to-granular performance gaps further indicate reliance on categorical heuristics rather than geometric reasoning. These results establish object orientation understanding as a fundamental open challenge for multimodal systems. Dataset: https://huggingface.co/datasets/appledora/DORI-Benchmark
Nazia Tasnim, Keanu Nichols, Yuting Yan +4
Boston University · Work done while at Boston University; currently at Johns Hopkins University
Layer-wise visual-text similarity in Multimodal Large Language Models (MLLMs) is widely interpreted as evidence that the language model progressively integrates visual content into a shared representation space. This reading rests on the assumption that scalar alignment scores reflect content-level cross-modal interaction. To test this assumption, we apply controlled interventions to the visual stream. Across 13 MLLMs from five families spanning 0.5B to 72B parameters, replacing projector-output visual tokens with Gaussian noise sharply reduces task accuracy, yet four standard scalar measures (CKA, SVCCA, MIR, and the leading principal-angle cosine) fail to consistently separate the corrupted stream from the original. We call this failure the alignment illusion and trace it to the shared language-model pathway: anisotropic MLP down-projections pull visual and text tokens toward common output directions, producing weight-induced alignment. Because this component is essentially one-dimensional, we introduce the principal-angle gap (PA gap), defined as the difference between the top two principal-angle cosines, which separates weight-induced similarity from multi-directional visual structure. Under graded visual corruption, the PA gap tracks task accuracy more consistently than the scalar scores we consider; under a structured but irrelevant image, it further exposes regimes in which internal geometry and task accuracy come apart. Internal visual-text alignment in MLLMs is therefore best read as a geometric diagnostic of the visual stream inside the language model rather than a direct proxy for content-level cross-modal interaction, and is most informative when calibrated by controlled task evidence.
Hong-Han Wang, Yuntao Wang, Hu Ding
University of Science and Technology of China, Hefei, China