LocalAgent: Collaborative Agentic Verification for Fine-Grained Instance-Level Consistency
Authors: Jiachen Guo, Xinshan Zhu, Boyun Wang, Yanyan Liang, Jun Wan, Feng Lin, Ajian Liu
Organizations: School of Electrical and Information Engineering, Tianjin University · SCSE, Macau University of Science and Technology (MUST) · AIX, University of Macau · MAIS, Institute of Automation, Chinese Academy of Sciences (CASIA) · State Key Laboratory of Blockchain and Data Security, Zhejiang University
Multimodal large language models (MLLMs) have made rapid progress in general visual understanding, yet they still struggle with fine-grained instance-level consistency verification, where the goal is to determine whether two images depict the exact same physical object under varying viewpoints, illumination, and backgrounds. This task requires both global semantic perception and precise examination of subtle local evidence, since semantically similar objects may differ only in small suspicious regions. To address this challenge, we propose LocalAgent, a collaborative agentic framework that integrates the complementary strengths of a generalist MLLM and lightweight specialist verifiers. The MLLM first reasons globally to identify suspicious local regions, which are then delegated to a Local Evidence Verifier (LEV) toolkit for fine-grained appearance and geometric matching. LEV comprises Local Appearance Consistency (LAC), which estimates appearance similarity under illumination and color changes, and Local Geometric Correspondence (LGC), which estimates spatial correspondence under viewpoint changes. LEV returns structured scalar evidence that the MLLM integrates for the final decision. We further introduce an auxiliary-capped reinforcement fine-tuning (RFT) strategy, in which answer correctness dominates auxiliary tool-use and format signals, enabling the policy to learn when verification is necessary. Experiments show that LocalAgent improves mean accuracy by 4.9% over the strongest CoT-RFT baseline while invoking a verifier on only 12.13% of samples and preserving the general multimodal capabilities of the underlying MLLM. The open-source repository is available at https://github.com/Firefly-zoo/LocalAgent-main.
Figures & tables
Figure 1: The motivation of LocalAgent. Standard MLLMs over-rely on global similarity and fail on instance-level verification in both directions. LocalAgent uses the MLLM as a spatial controller to localize suspicious regions, delegates them to the LEV toolkit (LAC, LGC) for invariant matching, and integrates the returned structured scalar evidence into the final decision.
Figure 2: Overview of the LocalAgent framework. Given an image pair, the MLLM performs initial global semantic perception and either commits to a direct answer or, when the assessment is inconclusive, acts as a spatial controller to localize suspicious regions. These regions are conditionally delegated to the LEV toolkit, which returns structured scalar evidence that the MLLM integrates with its initial reasoning state to produce the final prediction. The strategic verification policy is optimized via auxiliary-capped RFT.
Method
Mean
Pair
Mul
TWIN-Eval
ILIAS
Landmarks
MET
CUB
Inquire
P
M
T
P
M
T
P
M
T
P
M
T
P
M
T
P
M
T
Closed-source Models
GPT-4o †
86.1
91.1
81.1
92.2
71.2
81.7
95.0
95.5
95.3
87.3
74.7
81.0
86.3
72.9
79.6
91.7
86.0
88.9
94.3
86.3
90.3
Gemini2.5 Flash †
82.2
87.0
77.5
89.1
68.1
78.6
92.2
90.5
91.4
85.4
77.0
81.2
82.6
71.9
77.3
84.1
77.9
81.0
88.3
79.4
83.9
Open-source Baselines
PerceptionLM-3B
37.8
50.0
25.5
46.9
25.2
36.1
49.6
25.0
37.3
51.5
25.7
38.6
50.4
26.1
38.3
49.6
24.9
37.3
52.1
26.1
39.1
Table 1: Main quantitative results (Accuracy, %). P, M, and T denote Pair, Mul, and Total accuracy. The leading Mean, Pair, and Mul columns are the unweighted averages over the six data sources. † indicates closed-source models. Bold marks the best open-source result in each column.
Table 4Table 5
Figure 3: Qualitative results of LocalAgent on pairwise verification cases.
Table 7
Vision-Language
Text-only
Method
MME-RW-L
AI2D
SEED
ChartQA
TextVQA
MMMU
BLINK
MMLU
HellaSwag
GSM8K
Qwen3-VL-4B
29.03
84.20
73.05
75.00
70.35
77.20
65.8
68.80
86.90
89.54
LocalAgent
29.39
84.44
73.28
75.08
70.39
77.22
65.9
68.84
86.96
89.54
Table 8: General capability evaluation (%). LocalAgent matches Qwen3-VL-4B across vision-language and text-only benchmarks.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
TWIN-Eval subset
Base
CoT-RFT
LocalAgent
LEV invocation
Training-emphasis-aligned hard cases
72.1
76.7
80.9
43.8
Other cases
88.8
78.2
69.2
17.9
Appendix
Table 9: Stratified TWIN-Eval accuracy (%) by visual characteristics associated with the training emphasis, with LocalAgent’s LEV invocation rate per stratum.
Figure 4: Illustrative failure modes.
Setting
(δ,ϵ)
Mean
Default
(0.06, 0.12)
80.7
No auxiliary reward on incorrect answers
(0, 0)
79.5
Looser bounds
(0.10, 0.20)
77.1
Appendix
Table 10: Sensitivity of LocalAgent to the auxiliary bounds on incorrect predictions.
Split
95% bootstrap CI
McNemar p
Overall
[4.27, 5.70]
4.034×10−41
Pairwise
[3.28, 5.07]
7.849×10−20
Multiple
[4.67, 6.95]
5.181×10−23
Appendix
Table 11: Paired-bootstrap 95% confidence intervals and exact McNemar tests for the accuracy gain of LocalAgent over CoT-RFT.