Induced Numerical Instability: Hidden Costs in Multimodal Large Language Models
Authors: Wai Tuck Wong, Jun Sun, Arunesh Sinha
Organizations: School of Computing and Information Systems, Singapore Management University, Singapore · Management Science and Information Systems Department, Rutgers Business School, New Jersey, USA
The use of multimodal large language models has become widespread, and as such the study of these models and their failure points has become of utmost importance. We study a novel mode of failure that causes degradation in performance indirectly by optimizing a loss term that seeks to maximize numerical instability in the inference stage of these models. We apply this loss term as the optimization target to construct images that, when used on multimodal large language models, cause significant degradation in the output. We validate our hypothesis on state of the art models large vision language models (LLaVa-v1.5-7B, Idefics3-8B, SmolVLM-2B-Instruct) against standard datasets (Flickr30k, MMVet, TextVQA, VQAv2, POPE, COCO) and show that performance degrades significantly, even with a very small change to the input image, compared to baselines. Our results uncover a fundamentally different vector of performance degradation, highlighting a failure mode not captured by adversarial perturbations.
Figures & tables
Figure 1: We sample 300 input pairs, drawn uniformly from logarithmic space for multiplication and from linear space for addition, respectively, starting from the minimum and to the maximum positive values of a half-precision ( float16 ) number. We compare the absolute differences when we perform operations with both full and half precision as the data type We see that numerical imprecision generally increases as values increase for multiplication, meaning larger inputs values yield more numerical errors.
Figure 2: A two-layer tanh network illustrating how a small input perturbation δ produces amplified change at the output due to asymmetric activation saturation. When x1=x2=0 and x2 is perturbed slightly by δ , the output becomes y=−2tanh(4δ) . In this example the output around this region is around 8x the initial magnitude of the input perturbation, demonstrating local sensitivity at that region that is inherent within the network that cannot be resolved by increasing floating point precision.
Figure 3: Qualitative comparison of different perturbation types and their corresponding activation responses. The first row shows input images under four settings—clean, FGSM, PGD, and numerically unstable (ours) . The second row presents activation maps extracted from the first vision encoder layer of LLaVA-v1.5-7B. While adversarial perturbations (FGSM, PGD) introduce localized distortions, our numerically unstable input induces diffuse and misaligned attention, indicating degradation through a fundamentally different mechanism.
Figure 4: Examples of outputs under clean and numerically unstable inputs ( ε=16/255 ) for the Idefics3-8B model. Each column shows the perturbed input image, the associated question, and corresponding responses. The perturbed responses deviate semantically from both the ground truth and the model’s original answers on the clean image, despite nearly identical inputs. Additional examples for other models can be found in Appendix B .
Figure 5: Implementation-level numerical error on MSCOCO for Idefics3-8B. We plot accumulated absolute differences between float32 and float16 forward passes (summed over operations and samples) versus training epoch (optimizing Eqn. 9 ). Numerical error increases over epochs under our proxy loss.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Examples of outputs under clean and numerically unstable inputs ( ϵ=16/255 ) for the Janus-Pro-1B model. Each column shows the perturbed input image, the associated question, and corresponding responses. The perturbed responses deviate semantically from both the ground truth and the model’s original answers on the clean image, despite nearly identical inputs.
Figure 7: Examples of outputs under clean and numerically unstable inputs ( ϵ=16/255 ) for the LLaVa-1.5-7B model. Each column shows the perturbed input image, the associated question, and corresponding responses. The perturbed responses deviate semantically from both the ground truth and the model’s original answers on the clean image, despite nearly identical inputs.
Figure 8: Examples of outputs under clean and numerically unstable inputs ( ϵ=16/255 ) for the SmolVLM2-2.2B model. Each column shows the perturbed input image, the associated question, and corresponding responses. The perturbed responses deviate semantically from both the ground truth and the model’s original answers on the clean image, despite nearly identical inputs.
Figure 9: Training dynamics on the Numerical-COCO dataset across four multimodal models: LLaVA-v1.5-7B, Idefics3-8B, SmolVLM2-2.2B-Instruct, and Janus-Pro-1B. For each model, the left plot shows the supervised proxy loss used on the MSCOCO dataset, while the right plot shows the corresponding accumulated numerical-difference metric computed from forward-pass deviations against a high-precision reference.