Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models

Hulingxiao He; Zhi Tan; Yuxin Peng

cs.CVFeb 28, 2026

Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models

Authors: Hulingxiao He, Zhi Tan, Yuxin Peng

Organizations: Wangxuan Institute of Computer Technology, Peking University

Abstract

A high-performing, general-purpose visual understanding model should map visual inputs to a taxonomic tree of labels, identify novel categories beyond the training set for which few or no publicly available images exist. Large Multimodal Models (LMMs) have achieved remarkable progress in fine-grained visual recognition (FGVR) for known categories. However, they remain limited in hierarchical visual recognition (HVR) that aims at predicting consistent label paths from coarse to fine categories, especially for novel categories. To tackle these challenges, we propose Taxonomy-Aware Representation Alignment (TARA), a simple yet effective strategy to inject taxonomic knowledge into LMMs. TARA leverages representations from biology foundation models (BFMs) that encode rich biological relationships through hierarchical contrastive learning. By aligning the intermediate representations of visual features with those of BFMs, LMMs are encouraged to extract discriminative visual cues well structured in the taxonomy tree. Additionally, we align the representations of the first answer token with the ground-truth label, flexibly bridging the gap between contextualized visual features and categories of varying granularity according to user intent. Experiments demonstrate that TARA consistently enhances LMMs' hierarchical consistency and leaf node accuracy, enabling reliable recognition of both known and novel categories within complex biological taxonomies. Code is available at https://github.com/PKU-ICST-MIPL/TARA_CVPR2026.

Figures & tables

Figure 1: LMMs struggle with hierarchical visual recognition (HVR), failing to obey the hierarchical consistency on both known and novel categories.

Figure 2: Illustration of the training framework. Taxonomy-Aware Representation Alignment (TARA) is conducted alternately with No-Thinking RFT to improve the hierarchical recognition performance of LMMs with taxonomic knowledge absorbed from BFMs.

Base Model	RL	TARA	iNat21-Plant					iNat21-Animal
Base Model	RL	TARA	HCA	$\mathrm{Acc_{leaf}}$	POR	S-POR	TOR	HCA	$\mathrm{Acc_{leaf}}$	POR	S-POR	TOR
Qwen3-VL-2B	✗	✗	6.46	30.16	60.15	44.74	41.36	7.18	27.86	65.23	55.57	51.77
	✓	✗	9.23	31.96	64.58	50.81	47.86	8.57	29.32	66.98	57.84	54.10
	✓	✓	12.78	32.66	68.57	55.98	53.72	10.26	30.77	68.42	58.85	55.86
			+3.55	+0.70	+3.99	+5.17	+5.86	+1.69	+1.45	+1.44	+1.01	+1.76
Qwen2.5-VL-3B	✗	✗	10.89	39.73	65.77	48.90	48.24	16.70	40.26	73.03	64.05	60.80

Table 1: Effects of TARA on iNat-Plant and iNat-Animal datasets.

Species	RL	TARA	Order F1	Family F1
Known	✗	✗	17.16	10.83
	✓	✗	23.30	11.47
	✓	✓	41.56	25.47
			+18.26	+14.00
Novel	✗	✗	17.16	10.83
	✓	✗	23.30	11.47

Table 2: Effects of TARA on TerraIncognita dataset [ 8 ] .

Target Layer		iNat21-Plant
$\mathcal{L}_{\mathrm{V}}$	$\mathcal{L}_{\mathrm{T}}$	HCA	$\mathrm{Acc_{leaf}}$	POR	S-POR	TOR
/	/	6.46	30.16	60.15	44.74	41.36
14	/	10.72	33.27	65.35	51.40	49.11
14	14	10.40	33.43	65.27	51.14	49.26
14	28	12.78	32.66	68.57	55.98	53.72
28	14	11.87	33.20	67.17	54.62	51.88

Table 3: Ablation on target alignment layers of TARA.

Figure 3: Different designs of target alignment features. (a) and (b) are for $\mathcal{L}_{\mathrm{V}}$ , and (c)-(e) are for $\mathcal{L}_{\mathrm{C}}$ .

Features		iNat21-Plant
$\mathcal{L}_{\mathrm{V}}$	$\mathcal{L}_{\mathrm{T}}$	HCA	$\mathrm{Acc_{leaf}}$	POR	S-POR	TOR
(a)	(e)	10.30	32.22	65.50	52.36	49.59
(b)	(c)	12.01	33.62	67.39	55.12	52.25
(b)	(d)	11.59	33.41	67.18	53.50	51.53
(b)	(e)	12.78	32.66	68.57	55.98	53.72

Table 4: Ablation on target alignment features of TARA.

RL	TARA	Last	Avg.
✗	✗	13.30	85.00
✓	✗	14.40	84.40
✓	✓	18.30	85.90
		+3.90	+1.50

Table 5: Left: visual probing results on the last and averaged image hidden states. Right: evaluation results on ImageWikiQA [ 58 ] .

Figure 4: Qualitative comparison of No-Thinking RFT with and without TARA. The two columns show that TARA can achieve better leaf node accuracy and hierarchical consistency.

Figure 5: Training efficiency. Models trained with TARA achieve faster convergence.

Hierarchical Classification

Hierarchical classification uses relationships among labels to predict categories at multiple levels, from broad groups to fine-grained classes.

See all 89 papers

Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models
Jul 3, 2026Hulingxiao He, Zhi Tan, Yuxin PengHierarchical Classification Hierarchical Representation Learning
Wangxuan Institute of Computer Technology, Peking University
Hierarchy-Guided Multimodal Representation Learning for Taxonomic Inference
Mar 26, 2026Sk Miraj Ahmed, Xi Yu, Yunqi Li +2Hierarchical Representation Learning Multimodal Robustness
Computing and Data Sciences, Brookhaven National Laboratory, Upton, NY 11973, USA · Rutgers University, New Brunswick, NJ, USA
TaxonRL: Reinforcement Learning with Intermediate Rewards for Interpretable Fine-Grained Visual Reasoning
Mar 4, 2026Maximilian von Klinski, Maximilian SchallVisual Reasoning Fine-Grained Image Classification
Hasso Plattner Institute University of Potsdam, Germany

Jul 3, 2026J/K move · Enter open · S save

Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models

Hulingxiao He, Zhi Tan, Yuxin Peng

Wangxuan Institute of Computer Technology, Peking University

Hierarchical Classification Hierarchical Representation Learning Multimodal Large Language Models

Open paper AskPDF arXiv

Taxonomies provide key information about the semantic relationships between concepts and the inherent organization of vision and language. Despite their impressive capabilities, large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency. These models typically only rely on language modeling objectives during fine-tuning and lack explicit taxonomy-aware regularization. To address this, we propose Hierarchical Representation Regularization (

HiR^2

), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs. Specifically, we introduce a semantic-aware visual tree construction framework that extracts coarse-to-fine visual features from intermediate LLM layers guided by textual cues. The regularizer combines two complementary objectives: a taxonomic entailment loss that enforces hierarchy via hyperbolic entailment cones in the Lorentz model, and a discriminative dispersive loss that promotes angular separation of semantically similar embeddings on the unit sphere without disturbing the radial hierarchical structure. Extensive experiments demonstrate that

HiR^2

effectively captures taxonomic structures across diverse LMMs and fine-tuning methods. Code is available at https://github.com/PKU-ICST-MIPL/HiR2_ICML2026.

Abstract

Hierarchical Classification

Explore similar work

Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models

Hierarchy-Guided Multimodal Representation Learning for Taxonomic Inference

TaxonRL: Reinforcement Learning with Intermediate Rewards for Interpretable Fine-Grained Visual Reasoning