Continual Tool-Space Extension Learning for Multimodal Agents
Authors: Jinbae Seo, Jungin Park, Hyunsung Jang, Namkoo Ha, Jiyoung Lee, Kwanghoon Sohn
Organizations: Yonsei University · Daegu Gyeongbuk Institute of Science and Technology (DGIST) · LIG Defense & Aerospace · School of AI and Software, Ewha Womans University · Korea Institute of Science and Technology (KIST)
Multimodal agents largely rely on generic tools such as cropping, search, and code execution, leaving specialist vision models outside their tool space. However, bringing such a model into an already trained agent is not straightforward. The challenge is to learn each new tool efficiently while retaining the ability to use existing ones. This work proposes COTEX, continual tool-space extension learning, that adds new tools at efficient adapter-level cost. COTEX locates two sets of parameter directions; one triggers a tool call and thereby opens a multi-turn reasoning trajectory, while the other determines which tool is chosen. The adapter of a new tool is restricted to the calling directions that leave earlier tool choices unaffected by orthogonal constraint. We further introduce a protocol that converts specialist models' existing task-specific annotations into verified tool-use trajectories. Experiments on two representative agents and three specialist tools show gains on ten vision-language benchmarks, with at most a 0.3-point drop on existing tools.
Figures & tables
Figure 2: The overall training pipeline of COTEX.
Figure 3: Low-rank retention rate of tool-call gradient energy. Gradients of the tool-call loss are projected onto the top- r tool-sensitive directions identified by C(UΔℓ,Lcall) within the left singular-vector subspace of ΔWℓ , and the retained energy ratio Er/Etotal is averaged over layers. At r=64 , under 2% of the full rank, the projection retains 88% of the energy for DeepEyesV2 and 97% for Metis.
Method
VStar
HRbench4K
HRbench8K
TreeBench
MultihopSpatial
MC-Bench
Qwen2.5-VL
75.3
65.5
62.1
37.0
2.9
14.7
DeepEyesV2
81.7
76.6
71.3
43.2
12.4
38.4
+ Prompting
82.2
73.5
69.3
41.7
10.2
27.1
+ Finetuning
75.9
74.9
70.4
40.5
10.8
45.3
+ COTEX
83.3
76.3
71.4
43.5
17.4
50.6
Table 1: Performance comparisons on visual perception benchmarks, including VStar [ 28 ] and HRBench [ 26 ] , and spatial reasoning benchmarks, including TreeBench [ 24 ] , MultihopSpatial [ 13 ] , and MC-Bench [ 29 ] . The best and second-best scores in each column are shown in bold and underlined, respectively.
Method
SEED2Plus
TextVQA
OCRBench
BLINK
HB(3)
HB(4)
HB(5)
Qwen2.5-VL
68.8
72.0
80.6
47.6
37.1
29.0
15.3
DeepEyesV2
69.8
76.6
80.4
41.1
30.7
21.0
19.4
+ Prompting
69.3
76.7
80.1
62.1
38.7
27.4
26.6
+ Finetuning
69.4
76.7
80.9
58.1
43.5
32.3
25.0
+ COTEX
69.4
76.9
82.5
87.1
95.2
85.5
88.7
Table 2: Performance comparisons on text-rich VL benchmarks, including SEED2Plus [ 14 ] , TextVQA [ 22 ] , OCRBench [ 17 ] and depth perception benchmarks, including BLINK [ 6 ] , HardBLINK [ 2 ] . ‘HB(#points)’ denotes the number of comparison points in HardBLINK.
Figure 4: Distribution of tool calls on each benchmark before and after training with COTEX.
Figure 5: Qualitative examples of newly introduced tool capabilities. Each example illustrates the agent’s reasoning, tool invocation, returned observation, and final prediction. More examples are in Appendix.
Method
Stage
VStar
OCR
Detection
Depth
DeepEyesV2
Initial
81.7
–
–
–
Finetuning
+O
79.1
31.9
–
–
+G
75.9
41.6
50.6
–
+D
75.9
44.0
47.1
64.9
O-LoRA
+O
79.1
31.9
–
–
+G
75.4
39.8
50.6
–
Table 3: Comparison of adaptation methods under sequential tool-space extension. Each stage cumulatively introduces OCR (+O), detection (+G), and depth (+D). All values are accuracies (%).
Initial Agent
Method
VStar
HR4K
HR8K
DeepEyesV2
Initial
81.7
76.6
71.3
COTEX
83.3
76.3
71.4
Metis
Initial
90.6
79.0
78.6
COTEX
92.5
80.6
79.5
Table 4: Effect of before/after COTEX with different initial agents. Performance on other benchmarks is in Appendix.
Stage
VStar
OCR
Detection
Depth
Acc.
Call
Acc.
Call
Acc.
Call
Initial
81.7
–
–
–
–
–
–
+ SFT
80.6
42.2
28.9
35.6
32.2
93.5
100.0
+ RL
83.3
44.0
38.0
47.1
82.8
74.0
100.0
Table 5: Effectiveness on SFT and RL from the official DeepEyesV2 weights. Call denotes the tool invocation rate (%).
Agentic multimodal reasoning extends passive image understanding by allowing VLMs to actively acquire and refine visual evidence through visual tool interactions. Effective visual tool use requires three capabilities: grounding tool calls in visual context, composing tools across multiple steps, and adapting reasoning to tool-returned observations. However, existing approaches largely focus on grounding within fixed tool spaces and rigid invocation patterns, leaving composition and adaptation insufficiently addressed. We present VC-Tooler, which learns visual tool use as a compositional and adaptive capability. To this end, we first build a trajectory bank through a hierarchical synthesis pipeline covering three capability levels: single-tool grounding, multi-tool composition, and diverse tool contexts and interfaces. We then train the model in two stages: a supervised cold start that establishes these capabilities, followed by reinforcement learning that encourages accurate, efficient, and context-aware visual tool use. VC-Tooler achieves state-of-the-art performance among open-source models on both general-purpose and agentic benchmarks, including 95.8% on V* and 35.3% on VTC-Bench, and shows promising transfer under richer tool settings at inference time. Project page: https://w1zheng.github.io/VC-Tooler
Multimodal agents have achieved notable progress on complex reasoning tasks through tool use, yet remain limited by two issues: statically predefined tool inventories fail to generalize to unseen scenarios, and indiscriminate tool invocation incurs redundant cost and noise-induced errors. We propose MetaForge, a multimodal agent framework that learns when to invoke tools and how to evolve its toolset on demand. MetaForge factorizes agentic behavior into four coupled stages: Decide (judging whether tool use is warranted), Retrieve (selecting suitable tools), Adapt (grounding tool parameters in task context), and Forge (synthesizing new skills online and recycling them into the tool library for reuse), forming a closed judge-retrieve-adapt-forge-recycle loop. A unified orchestration policy enables the agent to choose among answering directly, reusing existing tools, or forging new ones. We jointly optimize invocation necessity, retrieval accuracy, execution effectiveness, and forged-skill reusability via reinforcement learning, with an explicit invocation-cost penalty discouraging redundant calls. Across 12 benchmarks, MetaForge consistently surpasses 16 baselines in accuracy, efficiency, and generalization, validating a paradigm shift from static tool inventories to on-demand self-evolution.
Shouang Wei, Houcheng Min, Xinpeng Dong +8
East China Normal University · Zhejiang University · Tsinghua University +2
Tool-augmented multimodal agents show strong benchmark gains, often taken as evidence that agents have learned to use tools. We argue that this interpretation can be premature: a tool-call trace alone does not show whether the tool supplied answer-critical information. We study two representative ``thinking with images'' agents, Thyme and DeepEyesV2, across real-world understanding, OCR, chart understanding, and mathematical reasoning. Each agent is compared with its Tool-Free counterpart and with a Pure-Text Reasoner trained from the same source pool without tool-calling trajectories. Tool access yields little consistent aggregate improvement, does not reliably reduce generated-token cost, and leaves only a small tool-only solved set: 93% of DeepEyesV2's tool-solved problems and 96% of Thyme's are also solved by at least one non-tool setting. Mechanism ablations further show that the full tool-use loop does not consistently outperform either the tool-call format or the returned execution result alone. In the settings we study, the analyzed agents appear to learn tool-calling patterns more reliably than tool-contributed capabilities, suggesting that evaluation should distinguish tool availability from whether tools actually expand what agents can solve.
Jiawei Guo, Donglei Yu, Yu Chen +6
University of Chinese Academy of Sciences · Amap, Alibaba Group · Tongyi Lab, Alibaba Group +1