Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning

Kunlun Xu; Yanqin Zhang; Wenwen Qiang; Jiahuan Zhou

cs.CVSep 12, 2026

Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning

Authors: Kunlun Xu, Yanqin Zhang, Wenwen Qiang, Jiahuan Zhou

Organizations: Wangxuan Institute of Computer Technology, Peking University, Beijing, China · Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Abstract

Multimodal Continual Instruction Tuning (MCIT) aims to exploit the incrementally accumulated knowledge to process multimodal inputs of diverse tasks, where parameter routing plays an important role. State-of-the-art methods rely on sample-to-task center similarity and cross-modal fusion with equal weight during routing. However, such solutions face two fundamental flaws: (1) Within each modality, the sample-to-task center distance is sub-optimal for routing since the abundant intra-task diversity information is underleveraged. (2) Different modalities exhibit varying reliability across tasks, where the modality with inter-task ambiguity can easily misguide the routing result. To address these problems, we propose Hyperbolic Uncertainty-aware Modality-Balanced Routing (Hyper-LLaVA) to improve parameter routing capacity based on cross-modality task feature uncertainty modeling. Specifically, to improve intra-modality task matching, Hyper-LLaVA accesses the sample-to-task distribution similarity in the Hyperbolic space. Besides, to alleviate the degradation brought by unreliable modalities, Hyper-LLaVA quantifies the task matching ambiguity within each modality to achieve adaptive balancing between task matching across modalities. Based on the complementary intra- and inter-modality task matching enhancement, our Hyper-LLaVA outperforms state-of-the-art approaches by large margins. Our source code is available at https://github.com/zhoujiahuan1991/ICML2026-Hyper-LLaVA

Figures & tables

Figure 1: (a) Within different modalities, the task feature regions may be ambiguous or distinct. The ambiguous task feature regions can often result in high erroneous task matching scores. and distinct task feature distribution may also generate low correct task matching scores. (b) Existing methods are vulnerable to erroneous task matching scores due to simply averaging the dual-modality predictions. (c) Our method aims to seek an adaptive balance between modalities to generate reliable task selection.

Figure 2: The overview of our proposed Hyper-LLaVA framework. The process consists of two stages: (Left) The Training Stage , comprising (a) standard instruction tuning with task-specific LoRA experts and (b) a Task-aware Modality Reliability Estimation module. This module quantifies task separability via Bhattacharyya Distance ( $D_{BC}$ ) to derive adaptive modality weights ( $\omega_{v},\omega_{t}$ ). (Right) The Inference Stage , specifically (c) the Hyperbolic Uncertainty-aware Modality-Balanced Routing. We map features into the Poincaré ball to capture task uncertainty and fuse modality similarities ( $S_{v},S_{t}$ ) using the learned weights for precise expert selection.

		Accuracy on Each Task (%)								Aggregate Results (%)
Method	Venue	SQA	TVQA	ImgNet	GQA	VizWiz	Gron	VQAv2	OVQA	MFN $\uparrow$	MAA $\uparrow$	MFT $\uparrow$	BWT $\uparrow$
Zero-shot	-	49.91	2.88	0.33	2.08	0.90	0.00	0.68	0.17	7.12	-	-	-
Multi-task	-	56.77	49.35	95.55	56.65	53.90	30.09	59.50	55.65	57.18	-	-	-
LwF	TPAMI 2017	74.45	49.70	39.30	52.00	50.45	7.05	62.25	47.80	47.88	59.71	68.13	-20.26
EWC	NAS 2017	75.20	55.36	67.50	54.70	52.90	15.40	64.45	35.05	52.57	61.63	65.59	-13.02
O-LoRA	CVPR 2023	76.90	42.65	15.85	40.25	45.10	0.30	54.35	54.00	41.18	56.28	56.99	-15.82

Table 1: Comparison of the proposed Hyper-LLaVA method with existing approaches in CoIN Benchmark.

		Accuracy on Each Task (%)						Aggregate Results (%)
Method	Venue	ImgNetR	ArxivQA	VizWiz	IconQA	CLEVR	Flickr	MFN $\uparrow$	MAA $\uparrow$	MFT $\uparrow$	BWT $\uparrow$
Zero-shot	-	16.27	53.73	38.39	19.20	20.63	41.88	31.68	-	-	-
Multi-task	-	91.67	90.83	57.87	78.43	76.63	61.72	76.19	-	-	-
LwF	TPAMI 2017	40.27	75.93	42.76	44.38	37.43	56.34	49.52	-	-	-
EWC	NAS 2017	39.05	77.88	43.24	45.33	39.72	55.94	50.20	-	-	-
O-LoRA	CVPR 2023	77.50	78.07	44.50	63.13	64.73	58.16	64.35	78.02	76.01	-13.99

Table 2: Comparison of the proposed Hyper-LLaVA method with existing approaches in UCIT Benchmark.

Metric Space	Fusion Strategy	MFN (%)
Euclidean	Static	56.75
Poincaré	Static	62.67 (+5.92)
Euclidean	Adaptive	66.42 (+9.67)
Poincaré	Adaptive (Ours)	66.62 (+9.87)

Table 3: Ablation study on the effectiveness of Hyperbolic Metric and Adaptive Fusion (metric: MFN).

Figure 3: Ablation study on modality fusion strategies.

Figure 4: Ablation study on modality fusion strategies (a), Task feature distributions in the Poincaré ball (b) and (c).

Figure 5: Visualization of model predictions across different tasks in comparison with state-of-the-art Hide ( Guo et al., 2025 ) .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Log-scale Inter-task Relevance Matrices on CoIN Benchmark

Figure 7: Log-scale Inter-task Relevance Matrices on UCIT Benchmark

Figure 8: Evolution of Task-aware Modality Weight

Figure 9: Visualization of model predictions across different tasks in comparison with state-of-the-art Hide ( Guo et al., 2025 ) .

Explore similar work

Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning
Aug 2, 2026Huiyu Yi, Yongqi Xu, Bogang Zhang +5Class-Incremental Learning Adaptive Model Routing
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China
CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
Jun 1, 2026Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi +1Mixture-of-Experts Models Multimodal Instruction Tuning
School of Artificial Intelligence, Nanjing University, China · State Key Laboratory of Novel Software Technology, Nanjing University, China
MAny: Merge Anything for Multimodal Continual Instruction Tuning
Apr 15, 2026Zijian Gao, Wangwang Jia, Xingxing Zhang +6Continual Learning for VLMs Multimodal Instruction Tuning
College of Computer Science and Technology, National University of Defense Technology · State Key Laboratory of Complex & Critical Software Environment · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology +1

Aug 2, 2026J/K move · Enter open · S save

Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning

Huiyu Yi, Yongqi Xu, Bogang Zhang, Dunwei Tu+4

School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China

Class-Incremental Learning Adaptive Model Routing Multimodal Instruction Tuning

Open paper AskPDF arXiv

Multimodal Continual Instruction Tuning (MCIT) enables multimodal large language models to acquire new tasks sequentially while retaining previously learned capabilities. Many recent methods maintain task-specific LoRA experts and route each input to one or more experts at inference. Yet the task-identification problem underlying expert routing remains under-explored. We show that routing is nearly saturated on widely used MCIT benchmarks. Textual fingerprints that leak task identity and short 4--10-task sequences with few competing experts jointly obscure the long-horizon routing problem. To expose this challenge, we introduce FLEX (Fingerprint-reduced Long-horizon Expert eXamination), a 34-task long-horizon MCIT benchmark with weakened textual fingerprints. FLEX groups tasks with similar instruction and answer formats but diverse visual and knowledge domains, normalizes their outer templates, and evaluates routing over a substantially larger expert pool. Crucially, we formulate progressive-LoRA routing as soft task-as-class Multimodal Class-Incremental Learning (MCIL): each task defines an incremental routing class, whose complete score distribution supplies the LoRA mixture weights, with hard routing as a discrete special case. FLEX exposes this expanding task-identification challenge, while the MCIL formulation provides a principled interface for transferring CIL methods to expert routing. We instantiate PureLoRA as a controlled baseline and adapt four CIL methods to four MCIT frameworks without modifying their LoRA experts or generation pipelines. Our plug-in routers improve strict LoRA matching by up to 16.3 percentage points and overall MacroScore by up to 4.6 points. Code is available at: https://github.com/RINC-CL/FLEX

Abstract

Explore similar work

Beyond Routing Saturation: A Long-Horizon Class-Incremental Perspective on Expert Routing in Multimodal Continual Instruction Tuning

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

MAny: Merge Anything for Multimodal Continual Instruction Tuning