cs.CVApr 14, 2026
SaveWhy MLLMs Struggle to Determine Object Orientations
Organizations: Department of Computer Science, Colorado State University, Fort Collins, CO, USA
Abstract
Multimodal Large Language Models (MLLMs) struggle with tasks that require reasoning about 2D object orientation in images, as documented in prior work. Tong et al. and Nichols et al. hypothesize that these failures originate in the visual encoder, since commonly used encoders such as CLIP and SigLIP are trained for image-text semantic alignment rather than geometric reasoning. We design a controlled empirical protocol to test this claim by measuring whether rotations can be recovered from encoder representations. In particular, we examine SigLIP and ViT features from LLaVA OneVision and Qwen2.5-VL-7B-Instruct models, respectively, using full images, and examine CLIP representations in LLaVA 1.5 and 1.6 using rotated foreground patches against natural background images. Our null hypothesis is that orientation information is not preserved in the encoder embeddings and we test this by training linear regressors to predict object orientation from encoded features. Contrary to the hypothesis, we find that orientation information is recoverable from encoder representations: simple linear models accurately predict object orientations from embeddings. This contradicts the assumption that MLLM orientation failures originate in the visual encoder. Having rejected the accepted hypothesis that MLLMs struggle with 2D orientation tasks because of visual encoder limitations, we still don't know why they fail. Although a full explanation is beyond the scope of this paper, we show that although present, orientation information is spread diffusely across tens of thousands of features. This may or may not be while MLLMs fail to exploit the available orientation information.
Figures & tables
| Image Set | Prompt |
| Whole Images | How much is the 2nd image rotated clockwise when compared to the first image? |
| In-Place Rotated Images | How much is the object(s) rotated clockwise in degrees when compared to the first image? |
| dog on beach | By how much is the dog in the center inside the circle rotated if it is known that the rotation is zero degrees when the dog’s legs are vertical? |
| lizard on fish | By how much is the lizard in the center inside the circle rotated if it is known that the rotation is zero degrees when the lizard is roughly horizontal with its tail pointing left and its head pointing right? |
| train on indoor | By how much is the train in the center inside the circle rotated if it is known that the rotation is zero degrees when the train is vertical with the train tracks at the bottom and the steam from the train going vertically upwards? |
Table 1: Image sets and corresponding prompts to LLaVA/Qwen2.5-VL-7B-Instruct.
Figure 1: Set of images - Sets A and B are used for experiments with LLaVA-OV and Qwen2.5-VL-7B-Instruct, and set C is used for LLaVA1.5 and LLaVA1.6. Since LLaVA-OV and Qwen2.5-VL-7B-Instruct can be prompted with 2 images, real images were used. But since LLaVA1.5 and 1.6 can be prompted with only 1 image, synthetic images with reduced artifacts were used to improve the prediction accuracy. [A: Whole Images] : (a) dog (b) train (c) lizard (d) beach (e) indoor (f) fish. [B: In-Place Rotated Objects] : (a) koala on beach (b) vase on indoor (c) vase and toaster on indoor. [C: Blended Images with foreground on background] : (a) dog on beach (b) lizard on fish (c) train on indoor environment, with foreground scales 1, 2 and 3 from left to right.
Figure 2: Collage of every 15th image from Sections A (a) dog (whole image) and B (a) koala (in-place rotated) of Figure 1 .
Figure 3: 2D orientation estimation performance comparison of LLaVA-OneVision and Qwen2.5-Vl-7B-Instruct on the images with the dog for 36 randomly selected images.
| Two Image Version Whole Images LLaVA-OV Qwen Instruct MAE Max Min MAE Max Min Dog 0.67 1.61 0.03 1.30 4.63 0.09 Lizard 0.44 1.63 0.01 2.20 6.31 0.06 Train 0.56 2.18 .01 2.43 10.01 0.29 Beach 0.74 2.67 0.03 1.63 4.98 0.03 Indoor 0.81 2.92 0.003 1.32 4.96 0.01 Fish 2.31 5.31 0.12 2.85 8.62 0.17 In-Place Rotated Object(s) Koala 0.36 1.47 0.01 1.01 3.66 0.06 Vase 0.48 1.76 0.003 0.90 4.18 0.01 Vase & Toaster 0.65 1.88 0.02 1.58 5.81 0.02 | Single Image Version Blended Images LLaVA 1.5 LLaVA 1.6 MAE Max Min MAE Max Min Dog Scale 1 1.41 4.19 0.06 0.62 2.40 0.001 Scale 2 0.89 3.20 0.0001 0.6 2.01 0.01 Scale 3 0.67 2.52 0.02 0.52 2.2 0.03 Train Scale 1 1.3 3.79 0.02 0.78 2.12 0.02 Scale 2 0.9 2.69 0.003 0.53 1.72 0.0002 Scale 3 0.72 2.12 0.008 0.42 1.2 0.005 Lizard Scale 1 2.08 7.24 0.0003 0.87 3.15 0.04 Scale 2 1.25 4.67 0.005 0.64 2.42 0.02 Scale 3 1.13 3.53 0.04 0.71 1.69 0.05 |
Table 2: LLaVA-OneVision , Qwen2.5-VL-7B-Instruct , LLaVA 1.5 and LLaVA 1.6 vision encoders estimate 2D orientation to a high degree of accuracy across diverse image sets.
| Two Image Version Whole Images LLaVA-OV Qwen Instruct K-S p-value K-S p-value Dog 0.11 0.69 0.12 0.6 Lizard 0.14 0.43 0.09 0.9 Train 0.11 0.74 0.11 0.7 Beach 0.14 0.44 0.11 0.78 Indoor 0.09 0.87 0.09 0.89 Fish 0.06 0.997 0.12 0.62 In-Place Rotated Object(s) Koala 0.08 0.94 0.12 0.61 Vase 0.08 0.97 0.16 0.32 Vase & Toaster 0.09 0.93 0.09 0.9 | Single Image Version Blended Images LLaVA 1.5 LLaVA 1.6 K-S p-value K-S p-value Dog Scale 1 0.09 0.51 0.05 0.99 Scale 2 0.07 0.81 0.08 0.7 Scale 3 0.1 0.46 0.12 0.37 Train Scale 1 0.06 0.95 0.06 095 Scale 2 0.07 0.9 0.09 0.56 Scale 3 0.08 0.75 0.1 0.41 Lizard Scale 1 0.06 0.96 0.08 0.74 Scale 2 0.08 0.68 0.04 1 Scale 3 0.06 0.95 0.08 0.69 |
Table 3: Low Kolmogorov–Smirnov (K-S) statistic coupled with -value alpha (0.05) means there’s no reason to reject the null hypothesis that the error distribution for LLaVA-OneVision , Qwen2.5-VL-7B-Instruct , LLaVA 1.5 and 1.6 are purely random (Gaussian).
Figure 4: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with dog. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian.
Figure 5: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with dog. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian.
| LLaVA OneVision | Qwen2.5-VL-7B-Instruct | ||
| Angle ( ∘ ) | Count | Angle ( ∘ ) | Count |
| 90 | 149 | 90 | 142 |
| 180 | 28 | 10-15 | 14 |
| 0 | 2 | 10 | 11 |
| 45 | 1 | 180 | 6 |
Table 4: Top LLaVA-OneVision and Qwen2.5-VL-7B-Instruct query responses for the 180 samples with the dog scene. Both models frequently respond that the 2D orientation is 90 ∘ .
Figure 6: Incremental feature substitution for LLaVA-OneVision on images with the dog scene. On the y axis, when y = 1, predicted value matches the target orientation and when y = 0, predicted value matches the anchor orientation. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 7: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the dog scene. On the y axis, when y = 1, predicted value matches the target orientation and when y = 0, predicted value matches the anchor orientation. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 8: Collage of every 20th image from the images with the dog foreground (biggest foreground)
Figure 9: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with dog for 36 randomly selected images
Figure 10: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with lizard for 36 randomly selected images
Figure 11: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with fish for 36 randomly selected images
Figure 12: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with train for 36 randomly selected images
Figure 13: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with indoor for 36 randomly selected images
Figure 14: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with koala for 36 randomly selected images
Figure 15: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with vase for 36 randomly selected images
Figure 16: 2D orientation estimation performance comparison between LLaVA OneVision and Qwen2.5-VL-7B-Instruct on the images with vase and toaster for 36 randomly selected images
Figure 17: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with dog foregrounds (scale 1) for 72 randomly selected images
Figure 18: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with dog foregrounds (scale 2) for 72 randomly selected images
Figure 19: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with dog foregrounds (scale 3) for 72 randomly selected images
Figure 20: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with lizard foregrounds (scale 1) for 72 randomly selected images
Figure 21: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with lizard foregrounds (scale 2) for 72 randomly selected images
Figure 22: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with lizard foregrounds (scale 3) for 72 randomly selected images
Figure 23: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with train foregrounds (scale 1) for 72 randomly selected images
Figure 24: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with train foregrounds (scale 2) for 72 randomly selected images
Figure 25: 2D orientation estimation performance comparison between LLaVA 1.5 and 1.6 on the images with train foregrounds (scale 3) for 72 randomly selected images
Figure 26: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with lizard. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 27: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with lizard. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 28: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with train. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 29: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with train. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 30: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with beach. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 31: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with beach. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 32: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 33: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 34: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with fish. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 35: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with fish. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 36: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with koala-beach. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 37: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with koala-beach. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 38: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with vase-indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 39: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with vase-indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 40: Statistical Analysis using visual plots for LLaVA-OneVision - results for images with vase-toaster-indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 41: Statistical Analysis using visual plots for Qwen2.5-VL-7B-Instruct - results for images with vase-toaster-indoor. Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 42: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with dog foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 43: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with dog foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 44: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with dog foregrounds (Scale 1). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 45: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with dog foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 46: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with dog foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 47: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with lizard foregrounds (Scale 1). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 48: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with lizard foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 49: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with lizard foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 50: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with lizard foregrounds (Scale 1). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 51: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with lizard foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 52: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with lizard foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 53: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with train foregrounds (Scale 1). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 54: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with train foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 55: Statistical Analysis using visual plots for LLaVA 1.5 - results for images with train foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 56: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with train foregrounds (Scale 1). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 57: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with train foregrounds (Scale 2). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
Figure 58: Statistical Analysis using visual plots for LLaVA 1.6 - results for images with train foregrounds (Scale 3). Together with the numerical results in Table 3 and the visual plots in this Figure, we can conclude that the residual error distribution (Table 2 ) is random/Gaussian
| LLaVA 1.5 | LLaVA 1.6 | ||
| Angle ( ∘ ) | Count | Angle ( ∘ ) | Count |
| 90 | 25 | 90 | 40 |
| not possible | 10 | not possible | 17 |
| 45 | 9 | 0 | 6 |
| 0 | 8 | no answer | 4 |
| no answer | 4 | 180 | 2 |
Table 5: LLaVA-LLaMA results for the 72 test samples with the beach scene in the background. LLaMA frequently responds that the 2D orientation is 90 ∘ or “not possible to determine”. The number of correct answers (among valid responses) is very low, even with high thresholds
Figure 59: Incremental feature substitution for LLaVA-OneVision on images with the lizard scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 60: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the lizard scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 61: Incremental feature substitution for LLaVA-OneVision on images with the train scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 62: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the train scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 63: Incremental feature substitution for LLaVA-OneVision on images with the beach scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 64: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the beach scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 65: Incremental feature substitution for LLaVA-OneVision on images with the indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 66: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 67: Incremental feature substitution for LLaVA-OneVision on images with the fish scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 68: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the fish scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 69: Incremental feature substitution for LLaVA-OneVision on images with the koala-beach scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 70: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the koala-beach scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 71: Incremental feature substitution for LLaVA-OneVision on images with the vase-indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 72: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the vase-indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 73: Incremental feature substitution for LLaVA-OneVision on images with the vase-toaster-indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 540,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 74: Incremental feature substitution for Qwen2.5-VL-7B-Instruct on images with the vase-toaster-indoor scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 128,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 75: Incremental feature substitution for LLaVA 1.5 on images with the beach background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 20,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 76: Incremental feature substitution for LLaVA 1.6 on images with the beach background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 16,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 77: Incremental feature substitution for LLaVA 1.5 on images with the fish background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 20,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 78: Incremental feature substitution for LLaVA 1.6 on images with the fish background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 16,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 79: Incremental feature substitution for LLaVA 1.5 on images with the indoor background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 20,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 80: Incremental feature substitution for LLaVA 1.6 on images with the indoor background scene. No matter how the features are selected (according to the magnitude of the weights in the regressor or the absolute difference between anchor and target feature values, or randomly). 16,000 features or more must be replaced to fool the predictor. (Note that the x-axis is the number of feature substitutions times .) This implies the orientation information is highly diffuse.
Figure 81: Results of foreground orientation estimation by LLaVA 1.5 for dog images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
Figure 82: Images with synthetic backgrounds used to test the impact of background rotations on foreground orientation estimation.
Figure 83: Results of foreground orientation estimation by LLaVA 1.6 for dog images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
Figure 84: Results of foreground orientation estimation by LLaVA 1.5 for lizard images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
Figure 85: Results of foreground orientation estimation by LLaVA 1.6 for lizard images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
Figure 86: Results of foreground orientation estimation by LLaVA 1.5 for train images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
Figure 87: Results of foreground orientation estimation by LLaVA 1.6 for train images (scale 1) when (a) only BG (b) BG and FG are rotated in the test images and training set consists of images with only FG rotated.
| S/N | Train Set | Test set | MAE (degrees) | |
| LLaVA 1.5 | LLaVA 1.6 | |||
| 1 | dog on chessboard (FG rotated) | baseline (FG rotated) | 1.1 | 0.71 |
| BG + FG rotated | 96.5 | 92.34 | ||
| BG rotated | 80.09 | 80.48 | ||
| dog on grid (FG rotated) | baseline (FG rotated) | 1.28 | 0.68 | |
| BG + FG rotated | 92.13 | 88.14 | ||
Table 6: Impact of background (BG) image rotation on foreground (FG) rotation (rot.) estimation for dog foreground images (Scale 1) using LLaVA 1.5 and LLaVA 1.6 - Mean Absolute Error (MAE) for synthetic background image sets under different rotation conditions. When background is rotated, performance: (1) degrades sharply when trained on only FG rot. images, (2) and (3) improves significantly when trained on BG+FG rot., (4) improves moderately when trained on only BG rot.
| S/N | Train Set | Test set | MAE (degrees) | |
| LLaVA 1.5 | LLaVA 1.6 | |||
| 1 | lizard on chessboard (FG rotated) | baseline (FG rotated) | 1.81 | 1.36 |
| BG + FG rotated | 85.2 | 85.55 | ||
| BG rotated | 80.2 | 79.93 | ||
| lizard on grid (FG rotated) | baseline (FG rotated) | 1.87 | 1.16 | |
| BG + FG rotated | 84.1 | 87.22 | ||
Table 7: Impact of background (BG) image rotation on foreground (FG) rotation (rot.) estimation for lizard foreground images (Scale 1) using LLaVA 1.5 and LLaVA 1.6 - Mean Absolute Error (MAE) for synthetic background image sets under different rotation conditions. When background is rotated, performance: (1) degrades sharply when trained on only FG rot. images, (2) and (3) improves significantly when trained on BG+FG rot., (4) improves moderately when trained on only BG rot.
| S/N | Train Set | Test set | MAE (degrees) | |
| LLaVA 1.5 | LLaVA 1.6 | |||
| 1 | train on chessboard (FG rotated) | baseline (FG rotated) | 1.2 | 0.77 |
| BG + FG rotated | 86.59 | 82.96 | ||
| BG rotated | 80.46 | 81 | ||
| train on grid (FG rotated) | baseline (FG rotated) | 1.3 | 0.94 | |
| BG + FG rotated | 87.4 | 87.96 | ||
Table 8: Impact of background (BG) image rotation on foreground (FG) rotation (rot.) estimation for train foreground images (Scale 1) using LLaVA 1.5 and LLaVA 1.6 - Mean Absolute Error (MAE) for synthetic background image sets under different rotation conditions. When background is rotated, performance: (1) degrades sharply when trained on only FG rot. images, (2) and (3) improves significantly when trained on BG+FG rot., (4) improves moderately when trained on only BG rot.