Continual Tool-Space Extension Learning for Multimodal Agents
Authors: Jinbae Seo, Jungin Park, Hyunsung Jang, Namkoo Ha, Jiyoung Lee, Kwanghoon Sohn
Organizations: Yonsei University · Daegu Gyeongbuk Institute of Science and Technology (DGIST) · LIG Defense & Aerospace · School of AI and Software, Ewha Womans University · Korea Institute of Science and Technology (KIST)
Multimodal agents largely rely on generic tools such as cropping, search, and code execution, leaving specialist vision models outside their tool space. However, bringing such a model into an already trained agent is not straightforward. The challenge is to learn each new tool efficiently while retaining the ability to use existing ones. This work proposes COTEX, continual tool-space extension learning, that adds new tools at efficient adapter-level cost. COTEX locates two sets of parameter directions; one triggers a tool call and thereby opens a multi-turn reasoning trajectory, while the other determines which tool is chosen. The adapter of a new tool is restricted to the calling directions that leave earlier tool choices unaffected by orthogonal constraint. We further introduce a protocol that converts specialist models' existing task-specific annotations into verified tool-use trajectories. Experiments on two representative agents and three specialist tools show gains on ten vision-language benchmarks, with at most a 0.3-point drop on existing tools.
Figures & tables
Figure 2: The overall training pipeline of COTEX.
Figure 3: Low-rank retention rate of tool-call gradient energy. Gradients of the tool-call loss are projected onto the top- r tool-sensitive directions identified by C(UΔℓ,Lcall) within the left singular-vector subspace of ΔWℓ , and the retained energy ratio Er/Etotal is averaged over layers. At r=64 , under 2% of the full rank, the projection retains 88% of the energy for DeepEyesV2 and 97% for Metis.
Method
VStar
HRbench4K
HRbench8K
TreeBench
MultihopSpatial
MC-Bench
Qwen2.5-VL
75.3
65.5
62.1
37.0
2.9
14.7
DeepEyesV2
81.7
76.6
71.3
43.2
12.4
38.4
+ Prompting
82.2
73.5
69.3
41.7
10.2
27.1
+ Finetuning
75.9
74.9
70.4
40.5
10.8
45.3
+ COTEX
83.3
76.3
71.4
43.5
17.4
50.6
Table 1: Performance comparisons on visual perception benchmarks, including VStar [ 28 ] and HRBench [ 26 ] , and spatial reasoning benchmarks, including TreeBench [ 24 ] , MultihopSpatial [ 13 ] , and MC-Bench [ 29 ] . The best and second-best scores in each column are shown in bold and underlined, respectively.
Method
SEED2Plus
TextVQA
OCRBench
BLINK
HB(3)
HB(4)
HB(5)
Qwen2.5-VL
68.8
72.0
80.6
47.6
37.1
29.0
15.3
DeepEyesV2
69.8
76.6
80.4
41.1
30.7
21.0
19.4
+ Prompting
69.3
76.7
80.1
62.1
38.7
27.4
26.6
+ Finetuning
69.4
76.7
80.9
58.1
43.5
32.3
25.0
+ COTEX
69.4
76.9
82.5
87.1
95.2
85.5
88.7
Table 2: Performance comparisons on text-rich VL benchmarks, including SEED2Plus [ 14 ] , TextVQA [ 22 ] , OCRBench [ 17 ] and depth perception benchmarks, including BLINK [ 6 ] , HardBLINK [ 2 ] . ‘HB(#points)’ denotes the number of comparison points in HardBLINK.
Figure 4: Distribution of tool calls on each benchmark before and after training with COTEX.
Figure 5: Qualitative examples of newly introduced tool capabilities. Each example illustrates the agent’s reasoning, tool invocation, returned observation, and final prediction. More examples are in Appendix.
Method
Stage
VStar
OCR
Detection
Depth
DeepEyesV2
Initial
81.7
–
–
–
Finetuning
+O
79.1
31.9
–
–
+G
75.9
41.6
50.6
–
+D
75.9
44.0
47.1
64.9
O-LoRA
+O
79.1
31.9
–
–
+G
75.4
39.8
50.6
–
Table 3: Comparison of adaptation methods under sequential tool-space extension. Each stage cumulatively introduces OCR (+O), detection (+G), and depth (+D). All values are accuracies (%).
Initial Agent
Method
VStar
HR4K
HR8K
DeepEyesV2
Initial
81.7
76.6
71.3
COTEX
83.3
76.3
71.4
Metis
Initial
90.6
79.0
78.6
COTEX
92.5
80.6
79.5
Table 4: Effect of before/after COTEX with different initial agents. Performance on other benchmarks is in Appendix.
Stage
VStar
OCR
Detection
Depth
Acc.
Call
Acc.
Call
Acc.
Call
Initial
81.7
–
–
–
–
–
–
+ SFT
80.6
42.2
28.9
35.6
32.2
93.5
100.0
+ RL
83.3
44.0
38.0
47.1
82.8
74.0
100.0
Table 5: Effectiveness on SFT and RL from the official DeepEyesV2 weights. Call denotes the tool invocation rate (%).