Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solution

Jiahong Liu; Ming Shen; Xiaohao Liu; Rex Ying; Menglin Yang; Tat-Seng Chua; Irwin King

cs.CVSep 24, 2026

Hyperbolic Multimodal Continual Learning: A Closest-Admissible Solution

Authors: Jiahong Liu, Ming Shen, Xiaohao Liu, Rex Ying, Menglin Yang, Tat-Seng Chua, Irwin King

Organizations: The Chinese University of Hong Kong · National University of Singapore · Yale University · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Existing continual-learning methods protect parameters, replayed examples, or Euclidean feature subspaces. When applied to hyperbolic multimodal models, they do not explicitly preserve the Lorentz geometry that jointly encodes within-modality similarity, cross-modal correspondence, and semantic hierarchy; sequential updates can therefore retain task scores while still distorting previously learned relations. We address this gap with Hyperbolic Multimodal Continual Learning (HMCL). We show that preserving the old multimodal geometry amounts to restricting all modalities to one shared hyperbolic isometry, which induces a family of admissible first-order parameter changes. We formulate a joint closest-admissible (CA) correction that retains the shared rotation best matching the candidate modal updates; its minimal-rotation (MR) special case fixes this rotation to zero. Both variants correct the displacement realized by AdamW, and task anchoring bounds within-task accumulation while preserving learning freedom. Across a unified 16-task classification-retrieval stream with three hyperbolic backbones, HMCL improves final performance and backward transfer over sequential fine-tuning and four continual-learning baselines; HMCL-CA gives the highest Overall score on every backbone. A modality-extended stream confirms the retrieval gains. Representation analyses find 81.2 to 95.5 percent less radial, angular, cross-modal, and paired-distance drift; ImageNet-WordNet results show better semantic ancestry and radial hierarchy.

Figures & tables

Figure 1: Continual adaptation of hyperbolic representations. Hyperbolic multimodal embeddings jointly encode task semantics and their hierarchy. Unconstrained sequential optimization may gradually distort this geometry and weaken structures learned from earlier tasks. Geometry-aware updates can reduce this form of forgetting.

Figure 2: Geometry of continual hyperbolic multimodal learning. Geodesic proximity encodes semantic similarity, while radial distance from the origin records specificity. Under our preservation conditions, all modalities undergo one common hyperbolic isometry. From the top-down view, first-order preservation of radial hierarchy limits a permitted update to the tangent direction.

Figure 3: Stage-wise relative task-performance change for HyCoCLIP-B on seed 42 in the unified 16-task stream. Columns correspond to the completed training stage $t$ , and rows correspond to evaluated task $i$ in stream order. Each observed cell reports $100(P_{i,t}-P_{i,i})/P_{i,i}$ , where $P_{i,i}$ is the score immediately after task $i$ is learned; thus, the outlined diagonal is zero, red denotes relative degradation, and blue denotes improvement. The gray upper triangle contains tasks not yet introduced. All panels share one color scale, the outlined rightmost column gives the final-stage changes, and (R) marks retrieval tasks evaluated with symmetric R@5; the remaining tasks use top-1 accuracy.

Figure 4: Fine-grained final retrieval performance on COCO and Flickr30K in the separate stream augmented with audio and thermal tasks. Each panel reports the mean over three paired seeds for one backbone; T2I and I2T denote text-to-image and image-to-text Recall@

\{1,5,10\}

, respectively. Gray, blue, and green bars denote Vanilla, HMCL MR , and HMCL CA . Appendix H.4 reports exact means and standard deviations. Higher values indicate better retrieval.

Figure 5: Dataset-wise drift for MERU-L (seed 1024) on the unified 16-task stream, measured from each task-end checkpoint to the final checkpoint using 50 fixed test anchors per old task. The 15 plotted rows exclude SST-2 because it is the final task and hence has no post-task drift interval. Gray circles, blue diamonds, and green squares denote Vanilla,

\mathrm{HMCL}_{\mathrm{MR}}

, and

\mathrm{HMCL}_{\mathrm{CA}}

, respectively; shaded rows identify retrieval tasks. Lower values indicate better preservation. In-panel percentages report each HMCL variant’s relative reduction in mean drift over the 15 plotted tasks compared with Vanilla.

Figure 6: Radial geometry of the complete 20-class ImageNet primate WordNet subtree for HyCoCLIP-B (seed 42). (a) Geometry immediately after learning ImageNet; (b–d) geometry at the final checkpoint of the 16-task stream. Final representations are globally aligned to the task-end reference by a proper spatial rotation before display, so the remaining change reflects radial and relational distortion rather than an arbitrary global orientation. Node color denotes WordNet depth, and edges show the fixed subtree relations. “rad.” is the mean absolute radial change normalized by the task-end mean radius; “pair.” is the mean absolute change in pairwise Lorentz distance normalized by the task-end mean distance. Both quantities are computed in the original representation space rather than from the two-dimensional drawing.

Panel of Figure 7 — Figure 7: Hierarchical concept retrieval along image-to-origin geodesics. For each Flickr30K example, we move from the image representation toward the manifold origin [ROOT] and display the nearest text concept along the resulting specific-to-generic path. HMCL gives more coherent sequences in these examples, such as hat $\to$ fashion $\to$ style , while Vanilla more often shifts to less related concepts after sequential training.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
Hyperbolic geometry and the Lorentz model
$d$	Intrinsic hyperbolic dimension; the Lorentz ambient space has dimension $d+1$ .
$K$	Squared hyperboloid radius ( $K>0$ ); the sectional curvature is $-1/K$ .
$\mathbb{H}^{d}_{K}$	$d$ -dimensional hyperbolic space with curvature $-1/K$ .
$\mathbf{G}$	Minkowski metric tensor, $\mathbf{G}=\operatorname{diag}(-1,1,\ldots,1)\in\mathbb{R}^{(d+1)\times(d+1)}$ .
$\langle\cdot,\cdot\rangle_{\mathcal{L}}$	Lorentzian inner product, $\langle\mathbf{x},\mathbf{y}\rangle_{\mathcal{L}}=\mathbf{x}^{\top}\mathbf{G}\mathbf{y}$ .

Appendix

Table 5: Summary of the notation used in this work.

Semantic Property	Geometric Quantity	Condition
$-$ Directly constrained
Semantic similarity	$d_{\mathcal{L}}(\mathbf{x},\mathbf{y})\propto\langle\mathbf{x},\mathbf{y}\rangle_{\mathcal{L}}$	(P1) , (P2)
Semantic specificity	$d_{\mathcal{L}}(\mathbf{x},\mathbf{o})\propto\\|\mathbf{x}_{[1:d]}\\|$	(P3)
$-$ Implied consequence (Proposition 2 )
Concept partial order	$\operatorname{aper}(\mathbf{x}),\operatorname{ext}(\mathbf{x},\mathbf{y})$	Implied by the conditions

Appendix

Table 6: Links between the desired semantic properties and the hyperbolic quantities that encode them.

Dataset	Task	Classes	Train	Test	Evaluation
FGVC Aircraft [ 78 ]	Classification	100	3,334	3,333	Accuracy
Caltech-101 [ 73 ]	Classification	102	2,448	6,084	Accuracy
ImageNet-1K [ 66 ]	Classification	1,000	1,281,167	50,000	Accuracy
CIFAR-10 [ 72 ]	Classification	10	45,000	10,000	Accuracy
CIFAR-100 [ 72 ]	Classification	100	45,000	10,000	Accuracy
CLEVR [ 81 ]	Classification	8	4,500	5,000	Accuracy

Appendix

Table 7: Composition and statistics of the 16-stage continual multimodal benchmark.

Figure 8: Modality-specific spatial-norm distributions after the complete MERU-L continual stream (seed 1024). The distributions pool 500 fixed test anchors from each of the 15 old tasks in the 16-task sequence; SST-2 is excluded only because it is last and has no post-task forgetting interval. Bars show final-checkpoint norms of old-task image and text embeddings, and color-matched dashed lines show their pooled task-end medians. The inset values report the median per-anchor ratio

\lVert\mathbf{z}_{\mathrm{final,space}}\rVert_{2}/\lVert\mathbf{z}_{\mathrm{task\text{-}end,space}}\rVert_{2}

Figure 9: Stage-wise stability and plasticity diagnostics replotted from the original CMCL analysis of Liu et al. [ 63 ] . At stage

t

, panel (a) reports the mean deviation of previously learned modality-pair alignment scores from their preceding reference values; deviations close to zero indicate stable old-pair alignment. For

t>1

, panel (b) reports the mean higher-order term used to assess the source plasticity condition

o(\eta)/\eta\leq 0

; nonpositive values satisfy that condition. These theorem-specific quantities are not BWT or task-score gains.

(a) COCO
Backbone	Method	T2I			I2T
		R@1	R@5	R@10	R@1	R@5	R@10
MERU-L	Vanilla	$9.303\pm 0.072$	$23.844\pm 0.065$	$32.835\pm 0.108$	$15.113\pm 0.114$	$33.633\pm 0.220$	$43.773\pm 0.247$
	HMCL MR	$8.848\pm 0.067$	$22.787\pm 0.183$	$31.955\pm 0.245$	$15.847\pm 0.121$	$34.573\pm 0.549$	$45.353\pm 0.186$
	HMCL CA	$\mathbf{10.725\pm 0.120}$	$\mathbf{26.557\pm 0.143}$	$\mathbf{36.344\pm 0.032}$	$\mathbf{18.040\pm 0.250}$	$\mathbf{37.293\pm 0.239}$	$\mathbf{47.913\pm 0.348}$
MERU-B	Vanilla	$9.491\pm 0.038$	$23.635\pm 0.113$	$32.550\pm 0.195$	$13.067\pm 0.522$	$29.293\pm 0.440$	$38.740\pm 0.596$

Appendix

Table 10: Fine-grained final retrieval performance in the separate protocol augmented with audio and thermal tasks (mean

\pm

standard deviation over three paired seeds). T2I and I2T denote text-to-image and image-to-text retrieval, respectively. The methods and configurations match Table 2 . Boldface marks the best result for each backbone and metric; shaded cells denote HMCL.

Figure 10: Clean ImageNet-1K training time at the third continual stage (seed 42). Each bar is one complete synchronized fit_task timing on a common RTX A6000 under the matched schedule; values are seconds. Jobs are serialized, and accepted runs contain no foreign GPU compute PID.

Explore similar work

Aug 10, 2026cs.LG

Hyperbolic Multimodal Continual Learning

Hyperbolic geometry has recently emerged as a powerful representation space for multimodal learning, as it naturally captures hierarchical semantic structure across modalities. Despite this progress, how such representations behave under continual learning poses fundamentally different challenges that remain underexplored. This work provides a geometric perspective on this problem and establishes a theoretical foundation for representation preservation in hyperbolic space, showing that preventing forgetting requires cross-modal invariance under a shared hyperbolic isometry. We further show that forgetting in hyperbolic continual learning involves both semantic relation drift and hierarchy-related distortion, motivating preservation of both cross-modal relational structure and hierarchical geometry. Guided by these insights, a principled continual learning framework is derived that preserves essential geometric structure while allowing effective adaptation to new tasks. Experiments on continual multimodal benchmarks corroborate the effectiveness of the proposed approach.

Jiahong Liu, Ming Shen, Xiaohao Liu +4

Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China · School of Computing, National University of Singapore, Singapore · Department of Computer Science, Yale University, New Haven, CT, United States +1

Hyperbolic Learning Replay-Based Continual Learning Continual Learning Multimodal Benchmarks Representation Quality

Ask PDF

Jul 2, 2026cs.AI

Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails

Multimodal large language models must continually adapt to evolving tasks and domains, yet standard continual learning metrics mainly measure whether old answers remain correct, leaving the stability of multimodal grounding largely unexamined. We study this overlooked failure mode and ask whether a continually adapted MLLM can preserve not only what it answers, but also how it uses visual, textual, OCR, chart, and document evidence. We identify \emph{hidden evidence-use forgetting}, where answer accuracy is retained while the model silently shifts toward different or less grounded evidence channels, and propose \textsc{RCL}, a replay-free reliance-constrained continual learning framework. \textsc{RCL} freezes the previous checkpoint as a behavioral reference, estimates teacher and student evidence-reliance profiles through counterfactual channel interventions, and jointly optimizes task learning, prediction preservation, and reliance preservation without adding inference-time cost. Across CoIN, COAST, MCITlib, and an evidence-sensitive multimodal stream, \textsc{RCL} consistently improves final performance and reduces forgetting over replay-free, PEFT, routing, and memory-assisted baselines, while substantially lowering modality reliance drift, dominant evidence flips, and hidden forgetting rates. These results suggest that robust continual multimodal learning requires preserving the evidence path behind correct answers, not merely the answers themselves.

Qianyu Chen, Canran Xiao, Runxuan Tang

1Nanyang Technological University · 2Shenzhen Campus of Sun Yat-sen University

Multimodal Learning Catastrophic Forgetting

Ask PDF

Jul 29, 2026cs.LG

Regularizing modality contribution drift in multimodal continual learning

Multimodal continual learning (MMCL) aims to learn emerging knowledge from multimodal data while preserving knowledge. To mitigate forgetting, current MMCL methods usually focus on cross-modal representation alignment or semantic similarity, but they overlook whether the relative contributions of individual modalities and their interactions remain stable across incremental tasks. We term this decision-level shift Modality Contribution Drift (MCD) and quantify it with the MCD score, which combines contribution-strength and relative-reliance changes under controlled interventions on modality subsets. Theoretical and empirical analyses further explain why current MMCL methods cannot reliably mitigate this drift. To this end, we propose Continual Modality Contribution Drift Regularization (CMCDR), which preserves the modality contribution structure of previously learned tasks. Since MMCL settings differ in whether old exemplars are available, CMCDR includes both replay-based and replay-free versions. The replay-based version uses modality-subset interventions as diagnostic probes on stored old samples, compares their contribution profiles between the current model and a frozen previous model, and constrains changes in old-sample modality-specific and interaction contributions. The replay-free version uses current-task samples as probes and distills the frozen model's old-task contribution responses, thereby regularizing the observed contribution profile without exemplars. Experiments on multimodal class-incremental learning and continual visual question answering validate the generality and effectiveness of CMCDR.

Zhen Zhang, Jielei Chu, Bin Liu +1

School of Computing and Artificial Intelligence, Southwest Jiaotong University

Multimodal Learning Continual Learning Contributions

Ask PDF