cs.CVSep 25, 2026

FLIP: Final Layer Inference-Time Probing for Vision-Language Models

Authors: Drandreb Earl O. Juanico, Rowel O. Atienza

Organizations: AI Graduate Program, University of the Philippines, Diliman, Quezon City, Philippines 1101 · EEE Institute, College of Engineering, University of the Philippines, Diliman, Quezon City, Philippines 1101

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present FLIP, a final-layer inference-time probe for testing whether a logit-facing intervention site in an open-weight vision-language model (VLM) supports structured, task-linked computation rather than generic perturbation. Behavioral change under internal intervention is otherwise mechanistically ambiguous: it may reflect improved use of visual evidence, generic output instability, or outright degradation. FLIP applies elementwise flooring to the final normalized hidden state before logit computation, leaving parameters, prompts, and decoding unchanged. On a controlled detection/counting probe, sweeping intervention strength reveals three regions: negligible change, a bounded interior regime in which detection recall at IoU 0.50 (R50R_{50}) improves while tolerant counting error (Ecount\mathcal{E}_{\mathrm{count}}) falls, and over-suppression. We formalize a four-criterion probe-and-sweep protocol for disciplining the interpretation of intervention effects: regime structure, grounding-proxy alignment, feature-coherence dependence, and failure to reproduce the same positive regime on a performance-based negative control. The post-normalization state passed to the output head is the logit-facing instantiation of this test; under a non-targeted flooring sweep it satisfies the full protocol. Raw decoder-layer interventions, including the last-block output before final normalization, and the singleton-pair left/right control fail to reproduce the Final-site signature, while same-site operators and multiple VLMs replicate it. FLIP is therefore a validation step for intervention-based mechanistic interpretability, not a steering method. Repo at https://github.com/earl-juanico/flip-qwen3vl/

More in this topic

VLM Interpretability

Vision-language model interpretability explains how image and language representations shape predictions, evidence use, and multimodal reasoning.

See all 268 papers

Explore similar work

CardsList
Oct 21, 2026

VLMProbe: An Open-source Library for VLM Probing and Interpretation

Understanding how vision-language models use visual information requires connecting internal representations to model behavior. We present vlmprobe, an open-source library that integrates lens readouts, attention attribution, causal interventions, and sparse autoencoder (SAE) analysis within a common model interface. Shared adapters, calibration and checkpoint utilities, evaluation pipelines, and visualizations support systematic analysis across architectures. Using this framework, we compare methods across vision-language, language, and vision models through matched case studies and controlled evaluations of pretrained sparse dictionaries. The comparisons show that representation readouts and spatial explanations depend on the analysis objective and observation site, and that local attribution estimates can disagree with measured intervention effects. SAE evaluations further expose distinct trade-offs in reconstruction fidelity, concept readout, and spatial localization. Together, these results support combining multiple forms of evidence to evaluate an interpretation. vlmprobe provides a reusable basis for selecting methods, comparing their conclusions, and testing hypotheses about model behavior. The code is available on github.com/ubc-tea/vlm-probe

Gexin Huang, Haisheng Fu, Yanting Yang +4

The University of British Columbia · Vector Institute

VLM InterpretabilitySparse AutoencodersAttention Intervention in VLMs
AskPDF
Sep 23, 2026

Looks the Same, Answers Differently: Flip-Direction Steering for Robust Vision-Language Reasoning

Vision-language models (VLMs) achieve strong visual reasoning performance, yet subtle changes from routine image capture and processing can alter their reasoning trajectories even when images appear nearly identical. In long-horizon generation, the resulting activation shifts may accumulate across decoding steps, progressively altering reasoning tokens and ultimately changing the final answer, a phenomenon referred to as answer flips. To address this instability, we propose FlipDir (Flip-Direction Steering), a training-free inference-time method that estimates a low-rank flip-inducing activation subspace from contrastive pairs of original and answer-flipping inputs and selectively steers hidden states during decoding. A margin-based gate limits subspace attenuation to uncertain decoding steps, recovering original predictions while preserving stable ones. To evaluate robustness beyond accuracy or consistency on fixed test sets, we introduce VisFlip, a benchmark framework that constructs evaluation groups for a target model and visual variation setting to separately assess recovery of original predictions and preservation of stable ones. VisFlip spans nine dataset-variation combinations across scientific reasoning, robot-scene understanding, and medical VQA, covering subtle visual variations common in each domain. Experiments across 18 settings demonstrate that FlipDir consistently outperforms existing methods on the combined recovery and preservation metric. We will make our code publicly available.

Yeonsung Jung, Joonhyun Jeong, Hoang Pham +4

Graduate School of AI, KAIST · NAVER Cloud · Department of Computer Science and Engineering, The Ohio State University +3

VLM RobustnessCounterfactual Evaluation of VLMsVLM ReasoningActivation Steering
AskPDF
Aug 16, 2026

Bit-Flip Attacks on Vision-Language-Action Models: Action-Decoding Architecture Shapes the Vulnerability

Quantized Vision-Language-Action (VLA) models expose a weight-fault surface: Rowhammer-style faults can corrupt deployed INT8 bits. We present the first bit-flip attack on a VLA: a few gradient-selected flips reduce closed-loop success to 0%0\%, while hundreds of random flips are harmless. Across four model variants spanning three action-head families, damaging bits concentrate in a few action-generating layers, but the empirical budget depends sharply on the head: direct regression and token policies fall in 11--55 flips, whereas the evaluated flow-matching policies require ∼100{\sim}100--300300. Our fixed-direction manifold-escape loss cuts \pizero{}'s budget from ∼1000{\sim}1000 to ∼100{\sim}100 flips, and a matched five-direction sweep shows that the attack is not specific to an all-positive direction. On a direct head, protecting 3.1%3.1\% of weights preserves 60%60\% success at K=100K{=}100, and protecting 5.3%5.3\% moves the open-loop break threshold from 3 to 100 flips. Finally, task-calibrated emulated K=100K{=}100 flips yield 0/200/20 real-robot successes, versus 14/2014/20 clean and 16/2016/20 global-random. Weight integrity is therefore a security boundary for embodied foundation models. Code is included as ancillary material.

Yudong Gao, Linghan Chen, Wenhan Wu +5

The Hong Kong University of Science and Technology · Adelaide University · Wuhan University +3

Adversarial AttacksAdversarial Attacks on VLA ModelsVision-Language-Action ModelsRobustness of VLA Models
AskPDF