LocalAgent: Collaborative Agentic Verification for Fine-Grained Instance-Level Consistency
Authors: Jiachen Guo, Xinshan Zhu, Boyun Wang, Yanyan Liang, Jun Wan, Feng Lin, Ajian Liu
Organizations: School of Electrical and Information Engineering, Tianjin University · SCSE, Macau University of Science and Technology (MUST) · AIX, University of Macau · MAIS, Institute of Automation, Chinese Academy of Sciences (CASIA) · State Key Laboratory of Blockchain and Data Security, Zhejiang University
Multimodal large language models (MLLMs) have made rapid progress in general visual understanding, yet they still struggle with fine-grained instance-level consistency verification, where the goal is to determine whether two images depict the exact same physical object under varying viewpoints, illumination, and backgrounds. This task requires both global semantic perception and precise examination of subtle local evidence, since semantically similar objects may differ only in small suspicious regions. To address this challenge, we propose LocalAgent, a collaborative agentic framework that integrates the complementary strengths of a generalist MLLM and lightweight specialist verifiers. The MLLM first reasons globally to identify suspicious local regions, which are then delegated to a Local Evidence Verifier (LEV) toolkit for fine-grained appearance and geometric matching. LEV comprises Local Appearance Consistency (LAC), which estimates appearance similarity under illumination and color changes, and Local Geometric Correspondence (LGC), which estimates spatial correspondence under viewpoint changes. LEV returns structured scalar evidence that the MLLM integrates for the final decision. We further introduce an auxiliary-capped reinforcement fine-tuning (RFT) strategy, in which answer correctness dominates auxiliary tool-use and format signals, enabling the policy to learn when verification is necessary. Experiments show that LocalAgent improves mean accuracy by 4.9% over the strongest CoT-RFT baseline while invoking a verifier on only 12.13% of samples and preserving the general multimodal capabilities of the underlying MLLM. The open-source repository is available at https://github.com/Firefly-zoo/LocalAgent-main.
Figures & tables
Figure 1: The motivation of LocalAgent. Standard MLLMs over-rely on global similarity and fail on instance-level verification in both directions. LocalAgent uses the MLLM as a spatial controller to localize suspicious regions, delegates them to the LEV toolkit (LAC, LGC) for invariant matching, and integrates the returned structured scalar evidence into the final decision.
Figure 2: Overview of the LocalAgent framework. Given an image pair, the MLLM performs initial global semantic perception and either commits to a direct answer or, when the assessment is inconclusive, acts as a spatial controller to localize suspicious regions. These regions are conditionally delegated to the LEV toolkit, which returns structured scalar evidence that the MLLM integrates with its initial reasoning state to produce the final prediction. The strategic verification policy is optimized via auxiliary-capped RFT.
Method
Mean
Pair
Mul
TWIN-Eval
ILIAS
Landmarks
MET
CUB
Inquire
P
M
T
P
M
T
P
M
T
P
M
T
P
M
T
P
M
T
Closed-source Models
GPT-4o †
86.1
91.1
81.1
92.2
71.2
81.7
95.0
95.5
95.3
87.3
74.7
81.0
86.3
72.9
79.6
91.7
86.0
88.9
94.3
86.3
90.3
Gemini2.5 Flash †
82.2
87.0
77.5
89.1
68.1
78.6
92.2
90.5
91.4
85.4
77.0
81.2
82.6
71.9
77.3
84.1
77.9
81.0
88.3
79.4
83.9
Open-source Baselines
PerceptionLM-3B
37.8
50.0
25.5
46.9
25.2
36.1
49.6
25.0
37.3
51.5
25.7
38.6
50.4
26.1
38.3
49.6
24.9
37.3
52.1
26.1
39.1
Table 1: Main quantitative results (Accuracy, %). P, M, and T denote Pair, Mul, and Total accuracy. The leading Mean, Pair, and Mul columns are the unweighted averages over the six data sources. † indicates closed-source models. Bold marks the best open-source result in each column.
Table 4Table 5
Figure 3: Qualitative results of LocalAgent on pairwise verification cases.
Table 7
Vision-Language
Text-only
Method
MME-RW-L
AI2D
SEED
ChartQA
TextVQA
MMMU
BLINK
MMLU
HellaSwag
GSM8K
Qwen3-VL-4B
29.03
84.20
73.05
75.00
70.35
77.20
65.8
68.80
86.90
89.54
LocalAgent
29.39
84.44
73.28
75.08
70.39
77.22
65.9
68.84
86.96
89.54
Table 8: General capability evaluation (%). LocalAgent matches Qwen3-VL-4B across vision-language and text-only benchmarks.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
TWIN-Eval subset
Base
CoT-RFT
LocalAgent
LEV invocation
Training-emphasis-aligned hard cases
72.1
76.7
80.9
43.8
Other cases
88.8
78.2
69.2
17.9
Appendix
Table 9: Stratified TWIN-Eval accuracy (%) by visual characteristics associated with the training emphasis, with LocalAgent’s LEV invocation rate per stratum.
Figure 4: Illustrative failure modes.
Setting
(δ,ϵ)
Mean
Default
(0.06, 0.12)
80.7
No auxiliary reward on incorrect answers
(0, 0)
79.5
Looser bounds
(0.10, 0.20)
77.1
Appendix
Table 10: Sensitivity of LocalAgent to the auxiliary bounds on incorrect predictions.
Split
95% bootstrap CI
McNemar p
Overall
[4.27, 5.70]
4.034×10−41
Pairwise
[3.28, 5.07]
7.849×10−20
Multiple
[4.67, 6.95]
5.181×10−23
Appendix
Table 11: Paired-bootstrap 95% confidence intervals and exact McNemar tests for the accuracy gain of LocalAgent over CoT-RFT.
Visual outcomes are increasingly central to multimodal large language models, making reliable and fine-grained verification essential for scaling generalist foundation models. In this work, we investigate multimodal meta-verification, which leverages verifier-generated rationales rather than decision-only signals, and explore how to effectively incorporate meta-verification feedback into multimodal verifier training. We identify two key findings. First, symbolic verifier outputs (e.g., bounding boxes) outperform textual explanations as meta-verification rationales, enabling efficient rule-based reinforcement learning rewards while avoiding reliance on model-based rewards from auxiliary judge models. Second, decoupling reinforcement learning objectives for binary judgment and meta-verification substantially outperforms joint reward optimization, due to intrinsic differences in output structure and learning dynamics. Based on these insights, we train OmniVerifier-M1, a generalist visual verifier leveraging symbolic meta-verification and decoupled reinforcement learning. OmniVerifier-M1 provides robust verification and fine-grained error localization, and further enables M1-TTS, a verifier-driven agentic generation system achieving dynamic region-level self-correction. This approach paves the way for more reliable, interpretable, and fine-grained multimodal verification, supporting safer and more controllable foundation model deployment.
Xinchen Zhang, Bowei Liu, Jiale Liu +7
Tsinghua University · Pennsylvania State University · University of Southern California +2
Multimodal Large Language Models (MLLMs) remain unreliable on fine-grained visual perception, even when high-resolution inputs preserve the necessary local details. We identify this limitation as visual context rot: decisive evidence may exist in the full image, yet fail to be reliably selected and used amid redundant visual context. We propose LOCUS (LOcal visual CUe Search), a training framework that teaches MLLMs to internalize local evidence search through a verifiable proxy task. During training, LOCUS provides a local crop as a visual cue and optimizes the model to recover its spatial support in the full image using an IoU-based reward. The visual cue is used only during training, leaving the standard image-question inference interface unchanged. Experiments across fine-grained perception, hallucination, general understanding, and reasoning benchmarks show that LOCUS improves localization-sensitive visual understanding while preserving broad capabilities. Attention analyses further indicate stronger focus on task-relevant evidence regions, suggesting that training-time visual cue search provides an effective route to internalized fine-grained evidence selection.
Zhou Tao, Fang Zhang, Zewen Ding +5
University of Science and Technology of China · State Key Laboratory of Cognitive Intelligence
Despite their popularity and success, Multimodal Large Language Models (MLLMs) often struggle to interpret images accurately, which limits their reasoning capability in complex scenarios (e.g., high object density and complex background clutter). Prior work mainly addresses this limitation by incorporating explicit visual cues like bounding boxes that require extra annotations. In addition, the resulting low-resolution crops often miss fine-grained details that MLLMs require for accurate reasoning. Therefore, we propose Mags-RL, an Agentic Reinforcement Learning (RL) framework that equips MLLMs with an external super-resolution "magnifying glass" agent for high-resolution fine-grained inspection. Specifically, the model performs two-round reasoning: in the first round, it generates an initial rationale and autonomously identifies regions of interest without relying on additional annotations; in the second round, it invokes a super-resolution agent to crop and upscale those regions, then revisits and verifies its earlier reasoning to produce the final answer. We also introduce a novel curriculum learning strategy that enables data-efficient RL training, needing as few as only 40 training samples to achieve reasonable performance. Experiments on VSR, TallyQA, and GQA subsets show its superior performance against recent strong competing methods, demonstrating high-quality reasoning with precise visual grounding. Code and weights will be released soon.
Xuanzhao Dong, Wenhui Zhu, Peijie Qiu +11
Arizona State University, Tempe, AZ, USA · Washington University in St. Louis, St. Louis, MO, USA · Clemson University, Clemson, SC, USA +3