Automatic Image-Level Morphological Trait Annotation for Organismal Images

Vardaan Pahuja; Samuel Stevens; Alyson East; Sydne Record; Yu Su

cs.CVApr 2, 2026

Automatic Image-Level Morphological Trait Annotation for Organismal Images

Authors: Vardaan Pahuja, Samuel Stevens, Alyson East, Sydne Record, Yu Su

Organizations: The Ohio State University · University of Maine

Abstract

Morphological traits are physical characteristics of biological organisms that provide vital clues on how organisms interact with their environment. Yet extracting these traits remains a slow, expert-driven process, limiting their use in large-scale ecological studies. A major bottleneck is the absence of high-quality datasets linking biological images to trait-level annotations. In this work, we demonstrate that sparse autoencoders trained on foundation-model features yield monosemantic, spatially grounded neurons that consistently activate on meaningful morphological parts. Leveraging this property, we introduce a trait annotation pipeline that localizes salient regions and uses vision-language prompting to generate interpretable trait descriptions. Using this approach, we construct Bioscan-Traits, a dataset of 80K trait annotations spanning 19K insect images from BIOSCAN-5M. Human evaluation confirms the biological plausibility of the generated morphological descriptions. We assess design sensitivity through a comprehensive ablation study, systematically varying key design choices and measuring their impact on the quality of the resulting trait descriptions. By annotating traits with a modular pipeline rather than prohibitively expensive manual efforts, we offer a scalable way to inject biologically meaningful supervision into foundation models, enable large-scale morphological analyses, and bridge the gap between ecological relevance and machine-learning practicality.

Figures & tables

Figure 1: Given an input specimen image, we first compute dense visual representations using an off-the-shelf backbone ( e.g. , DINOv2). These features are passed through a pre-trained sparse autoencoder (SAE), which identifies high-activation latent units corresponding to semantically meaningful regions (Algorithm 1 ). We extract the spatial masks associated with these activations and overlay them on the original image to localize trait-relevant boxes. Finally, a multimodal language model (MLLM) is prompted with the annotated image to generate fine-grained morphological trait descriptions. This results in a large-scale, automatically labeled image-level trait dataset.

C_{\text{species}}[s][z]=\sum_{i:y_{i}=s}\mathbf{1}[z\in\mathcal{Z}_{i}]\quad\quad C_{\text{genus}}[g][z]=\sum_{i:\text{genus}(y_{i})=g}\mathbf{1}[z\in\mathcal{Z}_{i}]

Algorithm 1 Salient Trait Extraction from Sparse Autoencoder Activations

Figure 2: Comparison of trait localization for Thymoites guanicae . Bioscan-Traits (left) generates interpretable trait descriptions that are tied to clear, specific anatomical structures. In contrast, Grad-CAM (center) produces diffuse heatmaps that highlight broad body areas without species-level disentanglement.

Figure 3: Comparison of salient morphological trait description generation using a just MLLM vs. MLLM + SAE ( $t_{\text{freq}}=1e-2$ ) for Agyneta straminicola . Each red box highlights a region selected by SAE neurons with high activation, indicating regions used for prompting the MLLM + SAE. The use of SAE helps MLLMs focus on salient morphological traits rather than general descriptions of all body parts.

Method	# Images	# Tokens	# Images	# Traits	Avg. Raw Rating	Avg. Rating
MLLM	1	$413$	–	–	$3.01$	$3.00$
MLLM	3	$940$	–	–	$3.12$	$3.15$
MLLM + SAE	$1$	$411$	$460$	$9,435$	$3.92$	$3.84$
MLLM + SAE	$3$	$1,072$	$460$	$7,897$	$4.01$	$3.91$

Table 1: Incorporating latent-specific patches significantly improves the quality of trait descriptions. Including multiple images in the prompt encourages MLLMs to focus on the traits common across all images, at the cost of more tokens per query. Using multiple images with SAE-extracted bounding boxes leads to improved precision, as better ratings indicate. We report both raw and mean-normalized ratings. The experimental setup uses Qwen2.5-VL-72B as MLLM, a normalized frequency threshold of (

t_{\text{freq}}

) =

3\text{\times}{10}^{-3}

, and

1,000

input images.

Figure 4: Comparison of salient morphological trait description generation using a single image vs. three images for Contacyphon ochraceus . Each red box highlights a region selected by SAE neurons with high activation, indicating regions used for prompting the MLLM + SAE. The use of multiple images yields a concise and taxonomically meaningful output, isolating traits with clearer morphological grounding.

Figure 5: Neurons 4852 and 13860 in SAE get activated at the wings and antennae of insects, respectively. The labels denote the highest annotated taxonomic level. Additional examples are shown in Appendix J .

Method	$\alpha$	$t_{\text{freq}}$	SAE MSE	SAE $L_{0}$	# Images	# Traits	Avg. Rating
MLLM+SAE	$2\text{\times}{10}^{-4}$	$1\text{\times}{10}^{-2}$	$8.8\text{\times}{10}^{-3}$	$1,081.1$	$60$	$60$	$3.84$
MLLM+SAE	$4\text{\times}{10}^{-4}$	$3\text{\times}{10}^{-3}$	$2.7\text{\times}{10}^{-2}$	$690.4$	$460$	$7,897$	$3.91$
MLLM+SAE	$4\text{\times}{10}^{-4}$	$1\text{\times}{10}^{-2}$	$2.7\text{\times}{10}^{-2}$	$690.4$	$20$	$20$	$3.58$
MLLM+SAE	$8\text{\times}{10}^{-4}$	$3\text{\times}{10}^{-3}$	$5.4\text{\times}{10}^{-2}$	$242.2$	$458$	$3,060$	$3.87$

Table 2: SAEs often trade off between reconstruction error (MSE) and sparsity (

L_{0}

). We investigate the effect of choosing between different balances of these errors. We find that lower sparsity performs better for both values of frequency threshold (

t_{\text{freq}}

). A lower value of the sparsity coefficient (

\alpha

) leads to lower MSE and thus better reconstruction. It improves the coverage of latents, leading to better recall. The experimental setup uses an input dataset of

1,000

images.

Task	Time	Remarks
Activation computation (1 image)	$2.74$ $\mathrm{ms}$	DINOv2 backbone
SAE forward (1 image)	$4.53$ $\mathrm{ms}$	Sparse Autoencoder
Total preprocessing (1 image)	$7.26\,\mathrm{ms}$	Feature extraction + SAE
MLLM inference (3 images / annotation)	$4.62\,\mathrm{s}$	Qwen2.5-VL-72B
Throughput (2 NVIDIA H100 80GB GPUs)	$208.9$ annotations/h/GPU

Table 4: Runtime and throughput of the proposed pipeline, measured on two NVIDIA H100 80GB GPUs. Times are averaged over the Bioscan-Traits workload.

Model	BioCLIP	BioCLIP 2
Baseline	$34.8$	$55.3$
Baseline + species-level fine-tuning ( Bioscan-Traits )	$39.6$	–
Baseline + trait-level fine-tuning ( Bioscan-Traits )	$39.9$	$56.23$

Table 5: Zero-shot species classification accuracy (%) on the Insects ( Ullah et al., 2022 ) benchmark. Incorporating trait-level supervision yields clear gains over the baseline pretrained model. BioCLIP 2 is pretrained on BIOSCAN-5M; therefore, we evaluate it directly under trait-level supervision.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Method	# Images	# Tokens	# Images	# Traits	Avg. Rating
MLLM	1	$413$	–	–	$3.00$ ( $\pm 0.71$ )
MLLM	3	$940$	–	–	$3.15$ ( $\pm 0.54$ )
MLLM + SAE	$1$	$411$	$460$	$9,435$	$3.84$ ( $\pm 0.63$ )
MLLM + SAE	$3$	$1,072$	$460$	$7,897$	$3.91$ ( $\pm 0.92$ )

Appendix

Table B.1: Incorporating latent-specific patches significantly improves the quality of trait descriptions. Including multiple images in the prompt encourages MLLMs to focus on the traits common across all images, at the cost of more tokens per query. Using multiple images with SAE-extracted bounding boxes leads to improved precision, as better ratings indicate. The experimental setup uses Qwen2.5-VL-72B as MLLM, a normalized frequency threshold of (

t_{\text{freq}}

) =

3\text{\times}{10}^{-3}

, and

1,000

input images.

Method	$\alpha$	$t_{\text{freq}}$	SAE MSE	SAE $L_{0}$	# Images	# Traits	Avg. Rating
MLLM+SAE	$2\text{\times}{10}^{-4}$	$1\text{\times}{10}^{-2}$	$8.8\text{\times}{10}^{-3}$	$1,081.1$	$60$	$60$	$3.84$ ( $\pm 0.70$ )
MLLM+SAE	$4\text{\times}{10}^{-4}$	$3\text{\times}{10}^{-3}$	$2.7\text{\times}{10}^{-2}$	$690.4$	$460$	$7,897$	$3.91$ ( $\pm 0.92$ )
MLLM+SAE	$4\text{\times}{10}^{-4}$	$1\text{\times}{10}^{-2}$	$2.7\text{\times}{10}^{-2}$	$690.4$	$20$	$20$	$3.58$ ( $\pm 1.05$ )
MLLM+SAE	$8\text{\times}{10}^{-4}$	$3\text{\times}{10}^{-3}$	$5.4\text{\times}{10}^{-2}$	$242.2$	$458$	$3,060$	$3.87$ ( $\pm 0.83$ )

Appendix

Table B.2: SAEs often trade off between reconstruction error (MSE) and sparsity (

L_{0}

). We investigate the effect of choosing between different balances of these errors. We find that lower sparsity performs better for both values of frequency threshold (

t_{\text{freq}}

). A lower value of the sparsity coefficient (

\alpha

) leads to lower MSE and thus better reconstruction. It improves the coverage of latents, leading to better recall. The experimental setup uses an input dataset of

1,000

images.

Method	MLLM	# Images	# Traits	Avg. Rating
MLLM	Qwen-2.5 VL 7B	$476$	$536$	$2.85$ ( $\pm 0.67$ )
MLLM	Qwen-2.5 VL 72B	$371$	$411$	$3.15$ ( $\pm 0.54$ )
MLLM + SAE	Qwen-2.5 VL 7B	$478$	$538$	$2.90$ ( $\pm 1.39$ )
MLLM + SAE	Qwen-2.5 VL 72B	$358$	$370$	$3.58$ ( $\pm 1.05$ )
MLLM + SAE	GPT-5 mini	$478$	$538$	$4.04$ ( $\pm 0.45$ )

Appendix

Table B.3: We investigate the effect of the verbalizer MLLM for morphological trait extraction for both the MLLM-only and MLLM + SAE models. We observe that GPT-5-mini achieves the highest average rating, outperforming both open Qwen-2.5-VL variants by a substantial margin. The larger Qwen-2.5-VL-72B model ( Wang et al., 2024 ) consistently obtains better ratings than its 7B counterpart. We note that GPT-5 mini and Qwen-2.5-VL-7B models might lead to false positives due to hallucination while extracting common traits in three input SAE-annotated images (Figure B.1 ). In contrast, the Qwen2.5-VL-72B model demonstrates improved robustness, avoiding such hallucinations and yielding more accurate trait descriptions. The experimental setup uses an input dataset of 20K images and

t_{\text{freq}}

1\text{\times}{10}^{-2}

Figure B.1: Comparison of morphological trait description quality between Qwen2.5-VL-7B, GPT-5 mini, and Qwen2.5-VL-72B for Diplonevra nitidula . Each red box highlights a region selected by SAE neurons with high activation, indicating regions used for prompting the MLLM + SAE. The Qwen2.5-VL-72B model correctly recognizes the background context and refrains from hallucinating visible traits, suggesting improved spatial grounding.

Figure C.2: Prompt for MLLM + SAE (multiple images)

Figure C.3: Prompt for MLLM + SAE (single image)

Figure C.4: Prompt for MLLM-only baseline (multiple images)

Figure C.5: Prompt for MLLM-only baseline (single image)

Hyperparameter	Value
Hidden Width	$24,576$ ( $32\times$ expansion)
Sparsity Coefficient $\alpha$	{ $2\text{\times}{10}^{-4}$ , $4\text{\times}{10}^{-4}$ , $8\text{\times}{10}^{-4}$ }
Sparsity Coefficient Warmup	$500$ steps
Batch Size	$16,384$
Learning Rate $\eta$	{ $5\text{\times}{10}^{-4}$ , $1\text{\times}{10}^{-3}$ }
Learning Rate Warmup	$500$ steps

Appendix

Table D.4: Hyperparameters for SAE training, filtering, and downstream fine-tuning.

Metric	Value
# Species	$736$
# Genera	$417$
# Unique images	$19.1$ K
# Samples	$80.8$ K

Appendix

Table D.5: Dataset statistics. On average, each image is associated with

4.2

trait samples.

Model	Top1 Accuracy (%)
CLIP ViT-B/16, $k$ NN	$24.57$
SigCLIP ViT-B/16, $k$ NN	$29.68$
DINOv2-base, $k$ NN	$41.28$

Appendix

Table E.6: Species classification accuracy on

1000

-species benchmark derived from BIOSCAN-5M (

20

train /

30

test images per species). DINOv2-base substantially outperforms CLIP ViT-B/16, using the kNN classifier.

Score	Example Image	Evaluation Criteria
5		Completely Correct — Body part correctly identified; all traits visibly match (color, texture, shape, size); no hallucinations. Example: “[Leg]: Thin, elongated, light brown, segmented.”
4		Mostly Correct — Body part is correct; Most traits are accurate, one minor imprecision. Example: “Leg: Thin, segmented, translucent; jointed structure” (leg is not translucent, but other traits are correct)
3		Partially Correct — Body part correct; 1–2 traits vague or incorrect. Example: “[Leg]: Thick, black, elongated.” (body part is correct, but leg is thin and brown.)
2		Mostly Incorrect — Incorrect body part or major trait mismatches. Example: “[Segmented]: All parts are visibly segmented” (the body part is missing, the segmented trait is correct though)
1		Completely Incorrect — Hallucinated or wrong body part. Example: “[Antennae]: dark brown.” (No antennae are visible)

Appendix

Table F.7: Example-based rubric for evaluating trait descriptions.

Figure G.6: Example 1 from Bioscan-Traits : “- Wing: Transparent, elongated, with visible veins. - Antenna: Thin, segmented, light brown”.

Figure G.7: Example 2 from Bioscan-Traits : “- [Leg]: Thin, elongated, light brown, segmented”.

Figure G.8: Example 3 from Bioscan-Traits : “- Wing: Transparent, elongated, with visible veins. - Antenna: Thin, segmented, dark brown”.

Figure G.9: Example 4 from Bioscan-Traits : “- Wing: Brown, translucent, folded, with visible veins”.

Figure G.10: Example 5 from Bioscan-Traits : “- Antenna: Thin, elongated, segmented, dark color”.

Model	Cost per annotation	Approx. cost for 100K images
Qwen2.5-VL-72B API (together.ai)	4.1\times 10^{-3}$ /annotation	$\$ 410$
GPT-5-mini API	8\times 10^{-4}$ /annotation	$\$ 80$

Appendix

Table H.8: Cost-of-use analysis for generating trait annotations with Qwen2.5-VL-72B (together.ai) and GPT-5-mini APIs, reported as average cost per annotation and extrapolated total cost for processing 100K images. The cost is averaged over the Bioscan-Traits workload, with

1,072

input tokens and

250

output tokens per annotation.

Figure J.11: Neurons 4040, 16584, 13433, and 14153 in SAE get activated at the thorax, at the place where the leg attaches to the body, eyes, and the abdomen, respectively. The labels denote the highest annotated taxonomic level.

Figure K.12: Comparison of salient morphological trait description generation using a just MLLM vs. MLLM + SAE for Scytodes intricata . Each red box highlights a region selected by SAE neurons with high activation, indicating regions used for prompting the MLLM + SAE.

Figure K.13: Comparison of salient morphological trait description generation using a just MLLM vs. MLLM + SAE for Erigone psychrophila . Each red box highlights a region selected by SAE neurons with high activation, indicating regions used for prompting the MLLM + SAE.

Figure K.14: Comparison of salient morphological trait description generation using a just MLLM vs. MLLM + SAE for Morulina thulensis . Each red box highlights a region selected by SAE neurons with high activation, indicating regions used for prompting the MLLM + SAE.

Figure K.15: Comparison of morphological trait description quality between Qwen2.5-VL-7B and Qwen2.5-VL-72B for Agyneta straminicola . Each red box highlights a region selected by SAE neurons with high activation, indicating regions used for prompting the MLLM + SAE. The larger model correctly identifies the highlighted body part of the insect.

Figure K.16: Comparison of morphological trait description quality between Qwen2.5-VL-7B and Qwen2.5-VL-72B for Erigone psychrophila . Each red box highlights a region selected by SAE neurons with high activation, indicating regions used for prompting the MLLM + SAE. The larger model correctly identifies the highlighted body part of the insect.

Figure K.17: Comparison of salient morphological trait description generation using a single image vs. three images for Deltocephalus fuscinervosus . Each red box highlights a region selected by SAE neurons with high activation, indicating regions used for prompting the MLLM + SAE. Using multiple images with SAE-extracted bounding boxes leads to dramatically improved precision.

Figure K.18: Comparison of salient morphological trait description generation using a single image vs. three images for Erigone arctophylacis . Each red box highlights a region selected by SAE neurons with high activation, indicating regions used for prompting the MLLM + SAE. Using multiple images with SAE-extracted bounding boxes leads to dramatically improved precision.

Sep 8, 2026

MorphoOrgaAgent: A Foundation-Model-Based Multi-Agent System for Autonomous Organoid Analysis

Organoids are three-dimensional tissue models whose morphology provides important insights into tumor development, disease progression, and drug testing. Extracting these morphological features relies heavily on manual segmentation, which is time-consuming and labor-intensive. Furthermore, performing quantitative statistical analysis typically requires custom coding skills and a mathematical background, presenting a major barrier for experimental biologists. To address these challenges, we introduce MorphoOrgaAgent, a multi-agent framework that achieves zero-shot organoid segmentation, automated data analysis, and report generation based on natural language input. The framework consists mainly of three core components: a TaskUnderstandingAgent that identifies requested measurements and visualization types; a hybrid segmentation module that combines Cellpose-derived geometric prompts with text prompts to guide SAM3 for zero-shot organoid instance segmentation; and a ReportAgent that computes quantitative metrics and compiles them alongside generated visualizations into a structured report. We further introduce MorphoOrgaVQA, a benchmark designed for quantitative evaluation of agent systems in organoid morphology analysis. Experimental results demonstrate that MorphoOrgaAgent handles both explicit and descriptive user requests, produces measurements closely matching ground truth, and generates complete analysis reports without requiring manual programming. The complete source code and MorphoOrgaVQA benchmark are publicly available at https://github.com/peng-lab/MorphoOrgaAgent.

Hanyi Zhang, Maximilian Hoermann, Lion J. Gleiter +5

Helmholtz AI, Helmholtz Munich - German Research Center for Environmental Health, Neuherberg, Germany · Department of Diagnostic and Interventional Radiology, University Hospital Heidelberg, Heidelberg, Germany · School of Computation, Information and Technology, Technical University of Munich, Munich, Germany +4

Image Segmentation Instance Segmentation Microscopy Image Analysis

Ask PDF

Jun 30, 2026

WildProp: Visual Estimation of Wildlife Body Proportions at Scale

Population-level morphometric measurements underpin ecological and evolutionary studies but traditionally require controlled imaging or physical specimen handling, limiting scalability. We present WildProp, a training-free framework that estimates wildlife body proportion distributions directly from large-scale, unconstrained image repositories. We cast morphometric estimation as a retrieval-driven correspondence problem: given a single user-annotated canonical image, WildProp performs pose-aware retrieval using foundation model features, transfers part endpoints via dense patch-level matching, filters predictions using geometric consistency, and aggregates measurements across retrieved images to estimate population-level ratio distributions. Unlike supervised keypoint pipelines, our approach adapts to arbitrary species and user-defined parts without per-species training. Evaluations on three large morphometric datasets spanning birds and amphibians show median relative errors of 10-20%. We further highlight the broad applicability of our approach through a number of case studies measuring various proportions across diverse taxa, including birds, frogs, insects, and flowers. Ablations demonstrate that pose-aware retrieval is critical for stable estimation, while robust aggregation mitigates keypoint and pose noise. Our results indicate that carefully curated 2D correspondences over web-scale imagery can provide scalable morphometric proxies for comparative and subgroup analyses across taxa, geography, and seasonality.

Mustafa Chasmai, Aaron Sun, Subhransu Maji

Manning College of Information and Computer Sciences University of Massachusetts, Amherst

Image Retrieval

Ask PDF

Jun 20, 2026

Morphology-Aware Multimodal Representation Learning for Insect Phylogenetic Reconstruction

Morphological traits provide important evidence for phylogenetic reconstruction and evolutionary relationship analysis. Recent image-based approaches have introduced deep learning, particularly convolutional models, to derive morphological features from specimen images, but these methods generally rely on single-modality visual representations and do not explicitly incorporate morphological semantics. This study proposes a morphology-aware multimodal alignment framework for insect phylogenetic reconstruction. The framework combines specimen images with curated morphological descriptions by adapting a vision transformer through parameter-efficient fine-tuning and supervised contrastive learning, followed by image-text alignment in a shared latent space. The learned image embeddings are then used as continuous traits for Bayesian phylogenetic reconstruction. On the public Rove-Tree-11 dataset, comparative and ablation experiments across multiple visual backbones and feature adaptation strategies demonstrate that multimodal alignment improves topological agreement with the reference phylogeny. The results indicate that the proposed framework can derive morphology-aware visual traits for computational phylogenetic reconstruction.

Zixuan Liu, Kaijie Yu, Chun He +5

Zhejiang Key Lab of Accessible Perception and Intelligent Systems, Zhejiang University, Hangzhou 310027, China · State Key Laboratory of Rice Biology and Breeding & Ministry of Agricultural and Rural Affairs Key Laboratory of Molecular Biology of Crop Pathogens and Insects, Institute of Insect Sciences, Zhejiang University, Hangzhou 310058, China · Rural Development Academy, Zhejiang University, Hangzhou 310058, China +1

Cross-Modal Alignment Phylogenetic Inference Multimodal Learning

Ask PDF

Abstract

Explore similar work

MorphoOrgaAgent: A Foundation-Model-Based Multi-Agent System for Autonomous Organoid Analysis

WildProp: Visual Estimation of Wildlife Body Proportions at Scale

Morphology-Aware Multimodal Representation Learning for Insect Phylogenetic Reconstruction