Vision-language agents equipped with tools expose intermediate outputs for inspection and comparison with the model's reasoning, which may offer safety benefits over single-shot models. This paper tests that potential benefit when the sensor reading itself is untrusted, so that the tool and reasoning channels share a common visual fault. We consider a dermatology agent based on segmentation, contrastive retrieval, and an iterative vision-language reasoner and define a coupling taxonomy that distinguishes single-channel errors from joint errors that agree or disagree. On PAD-UFES-20, we show that typographic injection produces detectable directional retrieval capture, whereas physical corruption does not. Crop-first retrieval preserves the payload at a readable scale and produces 55% joint wrong agreement across all cases (71% of faulty cases); full-image retrieval produces 35% (43%). Under a binary decision, if both channels are wrong, their predictions must agree, so hard-label channel comparison cannot detect joint failures, limiting disagreement recall to 0.29 (crop-first) versus 0.57 (full-image). The direction of the order effect is reproduced on the independent HIBA test set.
Table 2: Evaluation setup. Model weights are frozen.
Figure 2: The mechanism. (a) Directional capture Δcap with 95% patient-clustered bootstrap intervals: injection excludes zero, corruption does not. (b) Alignment rises across the 6–10 px payload-height transition. (c) Injection raises the mean non-cancer fraction among the top-5 neighbours by 0.37 (crop-first) and 0.10 (full-image).
Figure 3: System consequences. (a) Crop-first moves mass into joint wrong agreement (C4a). (b) Fault amplification ( CWR in Table 3 ); hollow markers mark intervals containing 1 . (c) Gate flag rate by disjunct (Sec. 4.5 ): channel comparison contributes nothing on C4a.
Perturb.
Arm
Bal. acc.
CWR
FAFcwr
none
single-shot
0.741
0.06
–
full-image
0.766
0.05
–
crop-first
0.772
0.05
–
injection
single-shot
0.384
0.29
–
full-image
0.270
0.38
1.43 [1.15, 1.79]
crop-first
0.247
0.44
1.70 [1.35, 2.13]
Table 3: Main results on the PAD-UFES-20 internal test (344 images, 206 patients).
Control
Rules out
Result
Neutral overlay
occlusion
Δcap 0.04 [ − 0.03, 0.11]
Physical corruption
non-directional shift
Δcap 0.02 [ − 0.05, 0.09]
Render-scale sweep
preprocessing artefact
transition at 6–10 px
Mask IoU
mask movement
0.94 / 0.91
Mask covers lesion
mask error
0.93 / 0.89
Biopsy-only subset
label provenance
FAF 1.66 [1.24, 2.19]
Table 4: Controls that could otherwise explain the order effect.
Quantity
Internal
External
Diff.
Bal. acc. (clean)
0.772
0.703
–
C4a, crop-first
0.55
0.49
−0.06 [ − .16,+.04]
C4a, full-image
0.35
0.31
−0.04 [ − .13,+.05]
FAFcwr , crop-first
1.70
1.58
−0.12 [ − .47,+.24]
Order gap
0.20
0.18
−0.02 [ − .14,+.10]
[95% CI]
[.12,.28]
[.09,.27]
Table 5: External validity: PAD internal test versus the HIBA clinical subset.
Agentic vision-language models (VLMs), which interleave textual reasoning with explicit tool calls such as cropping and code-based image manipulation, have emerged as a compelling paradigm for reliable and interpretable multi-modal reasoning. However, recent studies have revealed that such models often use tools unfaithfully. Many process images are irrelevant to the question (e.g., the crops miss the queried target), yet the tool call still receives full credit and the model still answers correctly. Such decorative or misaligned tool calls waste computation and reveal that the model does not faithfully use the evidence it retrieves. This may stem from two limitations of prevailing methods: the tool reward fails to distinguish useful from useless calls, and tool feedback carries no signal of usefulness. To this end, we introduce FaithEyes, a multi-agent self-judging framework. Concretely, we use a VLM to judge whether each process image helps answer the question. The judgement is injected into the reasoning context as part of the tool observation to help subsequent reasoning, and meanwhile is used to scale the tool reward by the helpful-tool ratio to suppress reward hacking. To keep judgement available at evaluation, we further design a multi-agent framework where the model itself serves as a subagent to judge the tool calls from the main agent, eliminating any dependence on external models at inference. Training via a two-stage SFT + RL pipeline on adapted open-source data, FaithEyes attains competitive or superior accuracy across visual perception and reasoning benchmarks, while substantially improving tool faithfulness and reducing inference cost. The homepage is at https://github.com/Mosi-AI/FaithEyes.
Haoqing Wang, Xingrun Xing, Ziheng Li +2
Samsung Research, Beijing, China · Peking University
Vision-language agents increasingly consume screenshots, documents, and user interfaces before writing to memory, sending messages, or invoking external tools. We study a concrete failure mode in this setting: action-boundary propagation, where sensitive or unsafe visible text is copied from an image into downstream tool arguments. We present VisualLeakBench, a diversified 500-image benchmark spanning UI, chat, document, form, and dashboard scenes, and evaluate a stratified 100-image agent subset with four production VLM systems under two workflows: note capture and external handoff. At baseline, target strings are propagated into tool arguments in 78.8% of PII cases and 85.5% of rendered unsafe-text cases. Under a defensive system prompt, rendered unsafe-text propagation remains high at 52.6%, while PII tool propagation falls to 2.0%, largely by suppressing tool use rather than preserving utility. Rates are tool-surface dependent: search-like tools suppress PII propagation, but rendered unsafe text still crosses tool boundaries. We measure visual-to-tool propagation rather than downstream instruction execution. We additionally provide a labeled-target oracle upper-bound diagnostic that localizes most failures at the tool boundary while leaving response-side leakage as residual risk.
Youting Wang, Yuan Tang, Yitian Qian +1
Northeastern University · Carnegie Mellon University · Boston University +1
Tool-integrated vision-language agents have made remarkable progress on compositional and multi-step visual reasoning. Yet their outputs frequently exhibit unfaithfulness: the stated reasoning path diverges from the computation that actually produced the answer, undermining reliability in safety-critical applications. We present DiffuseAgent-MI, a self-evolving agent whose perceptual grounding is governed by a KL-minimal energy model over feature units, providing a distributional view of visual mechanistic interpretability. The agent learns an energy landscape that softly constrains generated samples to lie near the native prior conditioned on the chosen interpretable unit, closing the gap between the explanation and the internal representation. A verifier then supplies trajectory-level faithfulness rewards, and a repair branch re-conditions the energy when the verifier flags an unfaithful step. On GeoQA, SciVis, VQA-v2 and an in-house multimodal reasoning set, DiffuseAgent-MI improves accuracy by up to 5.1 points over prior self-evolving agents while more than doubling mutual-information faithfulness and human-interpretability agreement. Our analysis shows the energy term and the verifier are complementary: the former guarantees distributional faithfulness, the latter trajectory-level faithfulness, and only their combination closes both gaps.
An Lanji, Dawei Liu, Jin Li +3
University of Electronic Science and Technology of China, Chengdu, China.