TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life

Mridul Khurana; Amin Karimi Monsefi; Justin Lee; Medha Sawhney; David Carlyn; Julia Chae; Jianyang Gu; Rajiv Ramnath

cs.CVMar 27, 2026

TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life

Authors: Mridul Khurana, Amin Karimi Monsefi, Justin Lee, Medha Sawhney, David Carlyn, Julia Chae, Jianyang Gu, Rajiv Ramnath,

Organizations: Virginia Tech · The Ohio State University · MIT · Boston University ·

Abstract

Accurately generating images across the Tree of Life is difficult: there are over 10M distinct species on Earth, many of which differ only by subtle visual traits. Despite the remarkable progress in text-to-image synthesis, existing models often fail to capture the fine-grained visual cues that define species identity, even when their outputs appear photo-realistic. To this end, we propose TaxaAdapter, a simple and lightweight approach that incorporates Vision Taxonomy Models (VTMs) such as BioCLIP to guide fine-grained species generation. Our method injects VTM embeddings into a frozen text-to-image diffusion model, improving species-level fidelity while preserving flexible text control over attributes such as pose, style, and background. Extensive experiments demonstrate that TaxaAdapter consistently improves morphology fidelity and species-identity accuracy over strong baselines, with a cleaner architecture and training recipe. To better evaluate these improvements, we also introduce a multimodal Large Language Model-based metric that summarizes trait-level descriptions from generated and real images, providing a more interpretable measure of morphological consistency. Beyond this, we observe that TaxaAdapter exhibits strong generalization capabilities, enabling species synthesis in challenging regimes such as few-shot species with only a handful of training images and even species unseen during training. Overall, our results highlight that VTMs are a key ingredient for scalable, fine-grained species generation.

Figures & tables

Figure 1: Existing text-to-image models struggle with fine-grained species details. We propose TaxaAdapter to inject VTM-derived embeddings into a frozen diffusion model to make fine-grained synthesis taxonomy-aware, enabling: (a) fine-grained species synthesis with accurate morphological details, (b) diverse and high-fidelity trait synthesis, (c) free-form text control, and (d) strong out-of-distribution generalization.

Figure 2: TaxaAdapter pipeline overview. Given a taxonomic name (e.g., Kingdom $\rightarrow$ Species ), we extract taxonomy-image aligned embeddings using a pre-trained vision taxonomy model (e.g., BioCLIP [ 5 ] , BioTrove-CLIP [ 44 ] or TaxaBind [ 35 ] ) and obtain complementary text features from the frozen CLIP text encoder. The dual conditioning streams are fused through a decoupled cross-attention mechanism, where the taxonomy branch captures species-level traits and the text branch retains free-form control over contextual cues such as style, background, or pose. During training, we only update the projection and cross-attention layers, while the diffusion backbone remains frozen for efficient and stable adaptation.

Figure 3: Caption-based trait fidelity evaluation. We leverage an MLLM to generate trait captions for real and generated images, summarize each set into a species-level trait description with an LLM, and compute text similarity between the two summaries. Our metric provides an interpretable, trait-level measure of morphological fidelity that complements standard image metrics.

Model	Image Quality		Classification Accuracy		Taxa2I Alignment		Caption-based Trait Fidelity
Model	FID $\downarrow$	LPIPS $\downarrow$	CAS@1 $\uparrow$	CAS@5 $\uparrow$	BioCLIP $\uparrow$	CLIP $\uparrow$	BERT $\uparrow$	CLIP $\uparrow$	ROUGE-L $\uparrow$	Mistral-7B $\uparrow$
SD 1.5 (ZS) [ 32 ]	66.91	0.779	1.09%	2.69%	15.52	18.95	0.28	0.80	0.24	0.88
SD 3.5-Large (ZS) [ 2 ]	70.98	0.802	2.12%	7.05%	16.19	19.36	0.31	0.81	0.26	0.90
FLUX.1-dev (ZS) [ 17 ]	86.33	0.808	2.82%	6.60%	15.76	19.41	0.34	0.83	0.29	0.91
SD 3.5-Large (LoRA [ 10 ] )	45.61	0.744	4.10%	13.33%	20.85	22.28	0.33	0.87	0.28	0.91
FLUX.1-dev (LoRA [ 10 ] )	69.63	0.775	2.92%	8.08%	17.13	20.10	0.31	0.85	0.28	0.91

Table 1: Results on iNat-mini over 156 species (following the evaluation protocol of [ 23 ] ). Numbers of TaxaDiffusion are directly taken from [ 23 ] ; note that classification accuracy and trait fidelity metrics are not available for their model. ZS denotes zero-shot results. We show TaxaAdapter results using two VTM text encoders, BioCLIP and TaxaBind. Best is boldened, second-best underlined.

Model	Image Quality		Classification Accuracy		Taxa2I Alignment		Caption-based Trait Fidelity
Model	FID $\downarrow$	LPIPS $\downarrow$	CAS@1 $\uparrow$	CAS@5 $\uparrow$	BioCLIP $\uparrow$	CLIP $\uparrow$	BERT $\uparrow$	CLIP $\uparrow$	ROUGE-L $\uparrow$	Mistral-7B $\uparrow$
SD 1.5 (ZS) [ 32 ]	46.89	0.7973	9.32%	25.54%	14.31	19.19	0.28	0.75	0.24	0.90
SD 3.5-Large (ZS) [ 2 ]	79.26	0.8179	11.00%	29.32%	14.25	19.43	0.28	0.74	0.24	0.90
FLUX.1-dev (ZS) [ 17 ]	90.09	0.8003	12.44%	31.14%	13.37	19.00	0.30	0.75	0.25	0.90
SD 3.5-Large (LoRA [ 10 ] )	64.21	0.7977	20.46%	50.36%	18.17	20.97	0.28	0.78	0.24	0.91
FLUX.1-dev (LoRA [ 10 ] )	59.13	0.7703	13.06%	34.70%	14.49	20.01	0.28	0.77	0.25	0.91

Table 2: Results on TreeOfLife -1M over 500 species.

Model	FID $\downarrow$	LPIPS $\downarrow$	BioCLIP $\uparrow$
SD 1.5 (ZS) [ 32 ]	61.93	0.7737	3.35
SD 1.5 (LoRA [ 10 ] )	43.91	0.7574	7.61
SD 1.5 (FT) [ 32 ]	39.41	0.7574	8.31
TaxaDiffusion [ 23 ]	31.87	0.7319	10.43
TaxaAdapter	30.68	0.7818	22.52

Table 3: Quantitative comparison on FishNet. Results are reported on 200 fish species following [ 23 ] . FT denotes full fine-tuning.

Figure 4: Qualitative comparison on TreeOfLife -1M. Each row shows a different species spanning birds, mammals, insects, and reptiles. TaxaAdapter generates morphology-faithful images that align with taxonomy-defining traits (e.g., texture patterns, body shape, coloration) while maintaining realistic textures and backgrounds. Notably, the first two rows illustrate two extremely similar species under the same Genus , where subtle differences in white spotting patterns on bird wings (highlighted in yellow circles) are correctly captured. Please see Appendix 0.C.5 for additional qualitative results.

	Single training image			Less than 5 training images
Model	FID $\downarrow$	LPIPS $\downarrow$	BioCLIP $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	BioCLIP $\uparrow$
SD 1.5 [ 32 ]	43.44	0.797	9.14	43.07	0.799	9.01
SD 3.5-Large [ 2 ]	68.66	0.817	8.46	71.42	0.819	8.39
FLUX.1-dev [ 17 ]	89.87	0.821	7.52	80.24	0.811	7.98
TaxaAdapter	32.19	0.733	18.63	35.43	0.724	17.78

Table 4: Long-tailed species generation results on ToL-1M. We report results for 5K species with only

1

(left) and fewer than

5

(right) training examples.

Figure 6: (a) Free-form text prompting. Our dual-conditioning design allows flexible control over contextual attributes while preserving species-level morphology. (b) Attention maps across different taxonomic tokens. TaxaAdapter gradually focuses on fine-grained information from body structure to head at Genus level to eyes and beak at Species level. (c) Results of different weighting factors $\lambda$ (Eq 5 ) during inference. Setting $\lambda=0$ relies solely on the CLIP text branch, while increasing $\lambda$ strengthens the taxonomy-conditioned route. Intermediate values (e.g., $\lambda=0.5$ ) balance both conditions and yield morphology-faithful generations.

Table 11

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Base model	Stable Diffusion v1.5 (frozen)
VTM encoder	BioCLIP / TaxaBind / BioTrove-CLIP (frozen)
Image resolution	$512\times 512$
Latent shape	$64\times 64\times 4$
Trainable parameters	$\approx 22$ M (proj + attn)
Epochs	100

Appendix

Table 8: Training and inference hyperparameter for TaxaAdapter. These settings are used across all datasets unless otherwise noted.

Model	BERT $\uparrow$	CLIP $\uparrow$	ROUGE-L $\uparrow$	Mistral-7B $\uparrow$
SD1.5	0.30	0.61	0.13	0.87
SD3.5-Large	0.28	0.61	0.13	0.87
FLUX.1-dev	0.30	0.62	0.13	0.87
TaxaAdapter	0.35	0.67	0.31	0.89

Appendix

Table 9: Quantitative results on instance-wise trait-fidelity for TreeOfLife -1M. Results are evaluated using 500 species on TreeOfLife -1M.

Model	BERT $\uparrow$	CLIP $\uparrow$	ROUGE-L $\uparrow$	Mistral-7B $\uparrow$
SD1.5	0.32	0.66	0.29	0.89
SD3.5-Large	0.32	0.66	0.29	0.89
FLUX.1-dev	0.34	0.69	0.30	0.90
TaxaAdapter	0.38	0.73	0.33	0.92

Appendix

Table 10: Quantitative results on instance-wise trait-fidelity for iNat-mini.

Panel of Figure 7 — Figure 7: Qualitative results for trait fidelity analysis using our captioning-based evaluation method. The upper case shows the trait captions generated using MLLM for real and generated images for species “ Sterna hirundo ” along with the summarized captions. The bottom case shows the trait-fidelity captions for species “ Amphispiza bilineata ”.

Figure 8: Different weighting factors $\lambda$ during inference for free-form text generation. Intermediate values (e.g., $\lambda=0.5$ ) balance both conditions and yield morphology-faithful generations. Note that the setting of this figure is different from Figure 6 -(c) in the main paper, where we show general generation results.

Figure 9: FID indicates species alignment; CLIP Score represents free-form text alignment (w.r.t. pose, background, etc.).

Model	FID $\downarrow$	LPIPS $\downarrow$	BioCLIP $\uparrow$
SD1.5	31.43	0.768	13.01
SD3.5-Large	70.63	0.806	11.93
FLUX.1-dev	74.14	0.801	11.62
TaxaAdapter	13.31	0.742	29.44

Appendix

Table 11: Additional Results in iNaturalist for 1100 species (one from each Family level in the taxonomy).

Figure 10: Synthetic “mixed or hybrid” species via taxonomic name mixing.

	Model	FID $\downarrow$	LPIPS $\downarrow$	BioCLIP $\uparrow$
(a)	SD1.5	55.38	0.769	13.57
(b)	SD1.5 (CLIP) + LoRA	24.74	0.742	17.49
(c)	SD1.5 (BioCLIP) + LoRA	74.60	0.742	10.08
(d)	TaxaAdapter	17.08	0.745	29.12

Appendix

Table 12: Analysis of Fine-tuning SD 1.5 with LoRA and also with replacing CLIP with BioCLIP. Models are trained on iNat-mini-birds (1500 species) and evaluated on 650 bird species.

SD backbone	FID $\downarrow$	LPIPS $\downarrow$	BioCLIP $\uparrow$
TaxaAdapter (SD1.5)	17.08	0.745	29.12
TaxaAdapter (SDXL)	13.39	0.743	29.54

Appendix

Table 13: Results of TaxaAdapter using different diffusion backbones on the iNat-mini birds subset. Both models use the same BioCLIP taxonomy encoder.

Model	FID $\downarrow$	LPIPS $\downarrow$	BioCLIP $\uparrow$	CAS@1 $\uparrow$	CAS@5 $\uparrow$
FLUX.1-dev (TN)	90.09	0.800	13.37	12.44%	31.14%
FLUX.1-dev (CN)	68.29	0.803	16.06	21.28%	42.42%
TaxaAdapter (TN)	29.87	0.734	26.69	61.98%	82.96%

Appendix

Table 14: Quantitative comparison on TreeOfLife -1M. Results are evaluated using 500 species on TreeOfLife -1M. For FLUX.1-dev, we show the difference in performance when using the taxonomic name vs the common name. TN: taxonomy name as prompt. CN: common name as prompt.

Figure 11: Additional qualitative results on TreeOfLife -1M.

Figure 12: Additional qualitative results on iNaturalist.

Figure 13: Additional qualitative results on FishNet.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

TreeAdapter: Hierarchical Taxonomy-Guided Adapter Composition for Fine-Grained Species Image Generation
Jul 27, 2026Yuze Sun, Zhongjie Duan, Yingda ChenHierarchical Representation Learning Image Generation
ModelScope Team, Alibaba Group
Fast and Useful: Adapting Distilled Generators for Visual Learning
Oct 23, 2026Patryk Bartkowiak, Dominik Michels, Sören Pirk +1Diffusion Model Adaptation Synthetic Data Augmentation
Adam Mickiewicz University · GreenMatterAI · KAUST +1
DeepTaxon: An Interpretable Retrieval-Augmented Multimodal Framework for Unified Species Identification and Discovery
Apr 27, 2026Jiawei Wang, Ming Lei, Yaning Yang +8Retrieval-Augmented Classification AI-Assisted Scientific Research
National University of Singapore · Hunan University · Xiangtan University +4

Jul 27, 2026J/K move · Enter open · S save

TreeAdapter: Hierarchical Taxonomy-Guided Adapter Composition for Fine-Grained Species Image Generation

Yuze Sun, Zhongjie Duan, Yingda Chen

ModelScope Team, Alibaba Group

Hierarchical Representation Learning Image Generation

Open paper AskPDF arXiv

Although general text-to-image models excel in open-domain generation, their performance degrades significantly in specialized downstream domains, particularly when generating images of rare biological species. Hindered by long-tailed distributions, general models struggle to capture subtle fine-grained details, while per-species fine-tuning methods over-isolate individual species and consequently ignore the shared visual features among closely related taxa. To address this, we propose TreeAdapter, a novel framework that explicitly leverages hierarchical taxonomic data. Rather than using a monolithic model or independent per-species modules, TreeAdapter attaches lightweight adapters to every node of the taxonomic tree. Specifically, leaf-node adapters capture species-specific visual traits, while internal-node adapters encapsulate shared semantics among descendant taxa. We introduce a two-stage training paradigm where ancestor adapters are optimized to model only the residual visual features unexplained by their descendants. This model architecture and training paradigm enable the model to fully leverage hierarchical information, ensuring the accurate generation of visual features for each species. Extensive experiments across three large-scale biodiversity benchmarks demonstrate that TreeAdapter achieves state-of-the-art fine-grained generation quality, outperforming both general-purpose and domain-specific baselines.

Abstract

Image Generation

Explore similar work

TreeAdapter: Hierarchical Taxonomy-Guided Adapter Composition for Fine-Grained Species Image Generation

Fast and Useful: Adapting Distilled Generators for Visual Learning

DeepTaxon: An Interpretable Retrieval-Augmented Multimodal Framework for Unified Species Identification and Discovery