Cross-modal learning for plankton recognition

Joona Kareinen; Veikka Immonen; Tuomas Eerola; Lumi Haraguchi; Lasse Lensu; Kaisa Kraft; Sanna Suikkanen; Heikki Kälviäinen

cs.CVMar 17, 2026

Cross-modal learning for plankton recognition

Authors: Joona Kareinen, Veikka Immonen, Tuomas Eerola, Lumi Haraguchi, Lasse Lensu, Kaisa Kraft, Sanna Suikkanen, Heikki Kälviäinen

Organizations: Lappeenranta-Lahti University of Technology LUT, Lappeenranta, Finland · Finnish Environment Institute, Helsinki, Finland · Faculty of Information Technology, Brno University of Technology, Czechia

Abstract

This paper considers self-supervised cross-modal coordination as a strategy enabling utilization of multiple modalities and large volumes of unlabeled plankton data to build models for plankton recognition. Automated imaging instruments facilitate the continuous collection of plankton image data on a large scale. Current methods for automatic plankton image recognition rely primarily on supervised approaches, which require labeled training sets that are labor-intensive to collect. On the other hand, some modern plankton imaging instruments complement image information with optical measurement data, such as scatter and fluorescence profiles, which currently are not widely utilized in plankton recognition. In this work, we explore the possibility of using such measurement data to guide the learning process without requiring manual labeling. Inspired by the concepts behind Contrastive Language-Image Pre-training, we train encoders for both modalities using only binary supervisory information indicating whether a given image and profile originate from the same particle or from different particles. For plankton recognition, we employ a small labeled gallery of known plankton species combined with a $k$ -NN classifier. This approach yields a recognition model that is inherently multimodal, i.e., capable of utilizing information extracted from both image and profile data. We demonstrate that the proposed method achieves high recognition accuracy while requiring only a minimal number of labeled images. Furthermore, we show that the approach outperforms an image-only self-supervised baseline. Code available at https://github.com/Jookare/cross-modal-plankton.

Figures & tables

Figure 1: Overview of the proposed multimodal plankton recognition model for CytoSense capable of classifying plankton species using (a) only bright-field images, (b) only optical profiles, or (c) both at the same time.

Panel of Figure 2 — Figure 2: Overview of the proposed multimodal recognition framework. (a) During pre-training, paired image–profile samples are aligned in a shared embedding space. (b) At inference, classification is performed using nearest-neighbor search in the embedding space.

Panel of Figure 3 — Figure 3: Example image and profile data

Dataset	Instrument	Number of classes	Samples per class	Total
LAB	CS-2019-97	24	543 – 1,285	20,050
SEA	CS-2019-97	38	1 – 2,594	9,353
UTO	CS-2021-103	unknown	unknown	32,930

Table 1: Summary of the datasets used in the experiments.

Panel of Figure 4 — Figure 4: Pre-processing and data augmentations: (a) original profile and image, (b) same pair pre-processed

Loss	Model	CNN	LSTM	Transformer
InfoNCE	EfficientNet B0	95.06 $\pm$ 0.58	89.59 $\pm$ 0.67	92.11 $\pm$ 0.98
	ConvNeXt Femto	94.81 $\pm$ 0.90	90.10 $\pm$ 0.67	89.85 $\pm$ 3.84
	ViT Tiny	96.01 $\pm$ 0.56	90.39 $\pm$ 0.66	91.80 $\pm$ 0.76

Table 2: In-domain average accuracy and standard deviation (%) for LAB

\rightarrow

LAB with I+P

\rightarrow

I+P

Gallery \| Test	I	P	I+P
I	92.73 $\pm$ 0.59	93.03 $\pm$ 0.69	94.24 $\pm$ 0.59
P	91.56 $\pm$ 0.67	94.51 $\pm$ 0.83	95.42 $\pm$ 0.73
I+P	92.79 $\pm$ 0.57	94.55 $\pm$ 0.82	96.01 $\pm$ 0.56

Table 3: Average accuracy and standard deviation for the best configuration (ViT-Tiny, CNN, InfoNCE) on LAB

\rightarrow

LAB

Model	LAB $\rightarrow$ SEA		UTO $\rightarrow$ LAB		UTO $\rightarrow$ SEA
	CNN	Transf.	CNN	Transf.	CNN	Transf.
EfficientNet B0	67.57 $\pm$ 2.23	66.87 $\pm$ 2.07	84.22 $\pm$ 0.99	88.35 $\pm$ 0.70	72.86 $\pm$ 1.85	69.04 $\pm$ 2.01
ConvNeXt Femto	68.05 $\pm$ 2.06	64.49 $\pm$ 3.42	83.73 $\pm$ 0.79	88.13 $\pm$ 0.74	70.63 $\pm$ 1.77	68.90 $\pm$ 2.27
ViT Tiny	66.75 $\pm$ 2.21	66.42 $\pm$ 1.90	86.81 $\pm$ 0.83	88.30 $\pm$ 0.71	71.75 $\pm$ 1.48	69.14 $\pm$ 1.75

Table 4: Cross-domain average accuracy and standard deviation (%) for different image and profile encoder combinations with I+P

\rightarrow

I+P

Model	CNN	Transformer
EfficientNet B0	71.05 $\pm$ 2.87	72.13 $\pm$ 2.03
ConvNeXt Femto	71.77 $\pm$ 2.42	71.67 $\pm$ 1.53
ViT Tiny	71.37 $\pm$ 2.17	72.73 $\pm$ 1.20

Table 5: Average accuracy and standard deviation (%) for combined training, UTO + LAB

\rightarrow

SEA with I+P

\rightarrow

I+P

Panel of Figure 5 — Figure 5: Average accuracy and standard deviation as a function of gallery set size, with $n$ being the number of multimodal samples in the gallery set (I+P $\rightarrow$ I+P)

Method	Model	LAB $\rightarrow$ LAB	LAB $\rightarrow$ SEA	UTO $\rightarrow$ LAB	UTO $\rightarrow$ SEA
DINO	EfficientNet B0	79.67 $\pm$ 1.42	57.30 $\pm$ 2.29	57.24 $\pm$ 0.81	63.19 $\pm$ 2.29
	ConvNeXt Femto	62.85 $\pm$ 2.82	55.52 $\pm$ 3.29	54.76 $\pm$ 0.87	55.58 $\pm$ 1.75
	ViT Tiny	59.54 $\pm$ 2.04	54.59 $\pm$ 1.83	53.24 $\pm$ 1.01	59.10 $\pm$ 2.42

Table 6: Average accuracy and standard deviation (%) comparing DINO and contrastive multimodal pre-training

Parameter	Value
Input size	6
Output size	257
Blocks	[2, 2, 2, 2]
Base channels	32
Dropout	0.2
Params	963K

Table 7: Configuration of the 1D CNN encoder.

Parameter	Value
Input size	6
Output size	257
Hidden size	128
N layers	1
Dropout	0.2
Params	264K

Table 8: Configuration of the Bidirectional LSTM encoder.

Parameter	Value
Input size	6
Output size	129
Hidden size	128
N heads	4
N layers	6
Feedforward Dim	256

Table 9: Configuration of the Transformer encoder.

EfficientNet B0 + CNN
Gallery	Test	Accuracy
I	I	89.76 $\pm$ 0.65%
I	P	90.95 $\pm$ 0.80%
I	I+P	92.01 $\pm$ 0.68%
P	I	89.05 $\pm$ 0.83%
P	P	93.93 $\pm$ 0.69%

Table 10: Full results, LAB

\rightarrow

LAB, InfoNCE

EfficientNet B0 + CNN
Gallery	Test	Accuracy
I	I	80.58 $\pm$ 1.32%
I	P	84.41 $\pm$ 1.37%
I	I+P	83.52 $\pm$ 1.29%
P	I	79.55 $\pm$ 1.20%
P	P	90.79 $\pm$ 0.63%

Table 11: Full results, LAB

\rightarrow

LAB, Sigmoid

EfficientNet B0 + CNN
Gallery	Test	Accuracy
I	I	63.47 $\pm$ 2.47%
I	P	29.35 $\pm$ 3.80%
I	I+P	64.25 $\pm$ 2.72%
P	I	28.58 $\pm$ 4.75%
P	P	60.31 $\pm$ 2.21%

Table 12: Full results, LAB

\rightarrow

SEA, InfoNCE

EfficientNet B0 + CNN
Gallery	Test	Accuracy
I	I	69.87 $\pm$ 1.21%
I	P	17.54 $\pm$ 2.17%
I	I+P	69.81 $\pm$ 1.21%
P	I	16.35 $\pm$ 1.20%
P	P	80.01 $\pm$ 0.95%

Table 13: Full results, UTO

\rightarrow

LAB, InfoNCE

EfficientNet B0 + CNN
Gallery	Test	Accuracy
I	I	71.04 $\pm$ 2.01%
I	P	31.37 $\pm$ 3.94%
I	I+P	71.69 $\pm$ 2.14%
P	I	21.49 $\pm$ 2.01%
P	P	64.73 $\pm$ 2.15%

Table 14: Full results, UTO

\rightarrow

SEA, InfoNCE

EfficientNet B0 + CNN
Gallery	Test	Accuracy
I	I	68.46 $\pm$ 2.94%
I	P	30.21 $\pm$ 3.37%
I	I+P	68.44 $\pm$ 3.23%
P	I	37.15 $\pm$ 4.03%
P	P	64.07 $\pm$ 2.92%

Table 15: Full results, LAB + UTO

\rightarrow

SEA, InfoNCE

EfficientNet B4 + CNN
Gallery	Test	Accuracy
I	I	73.60 $\pm$ 2.23%
I	P	36.73 $\pm$ 3.04%
I	I+P	73.99 $\pm$ 2.11%
P	I	38.81 $\pm$ 3.51%
P	P	65.19 $\pm$ 2.31%

Table 16: Full results, LAB + UTO

\rightarrow

SEA, InfoNCE

Cross-Modal Representation Learning

Cross-modal representation learning connects information across data types such as images, text, audio, biosignals, and sensor measurements.

See all 210 papers

May 22, 2026

Planktonzilla: Multimodal dataset and models for understanding plankton ecosystems

Marine plankton underpin aquatic food webs and play a key role in global CO2 sequestration, making reliable species identification critical for understanding ocean health and climate feedbacks. Existing classification models perform well on individual collections but fail to generalize across instruments and environments due to isolated training datasets and inconsistent labels. To address this, we introduce Planktonzilla-17M, a unified dataset consolidating publicly available plankton image collections spanning thirteen imaging systems. It comprises 17.4 million images with standardized taxonomy and geo-environmental metadata, including 3.74 million plankton images spanning over 602 taxonomic classes, of which 201 are identified at the species level, making it the largest and most comprehensive plankton image dataset to date. Using this large-scale dataset, we perform a controlled comparison between supervised and CLIP-style image--text training on a shared ViT backbone. We find that a supervised classifier matches or exceeds CLIP-style training when trained using taxonomic lineage as text. We further observe that BioCLIP and BioCLIP2 perform poorly on plankton in zero-shot and few-shot settings. Leveraging Planktonzilla-17M improves plankton classification performance, highlighting the limitations of current biological foundation models in marine imaging domains.

Alan Gerson Contreras Montanares, Luis Valenzuela, Luis Martí +1

Inria Chile Research Center · Inria — Institut national de recherche en sciences et technologies du numérique · Av. Apoquindo 2827, Las Condes 7550268 Chile.

Vision-Language Models Vision Foundation Models Image Classification

Ask PDF

Apr 20, 2026

PlankFormer: Robust Plankton Instance Segmentation via MAE-Pretrained Vision Transformers and Pseudo Community Image Generation

Plankton monitoring is essential for assessing aquatic ecosystems but is limited by the labor-intensive nature of manual microscopic analysis. Automating the segmentation of plankton from crowded images is crucial, however, it faces two major challenges: (i) the scarcity of pixel-level annotated datasets and (ii) the difficulty of distinguishing plankton from debris and overlapping individuals using conventional CNN-based methods. To address these issues, we propose PlankFormer, a novel framework for plankton instance segmentation. First, to overcome the data shortage, we introduce a method to generate labeled Pseudo Community Images (PCI) by synthesizing individual plankton images onto diverse backgrounds, including those created by generative models. Second, we propose a segmentation model utilizing a Vision Transformer (ViT) backbone with a Mask2Former decoder. To robustly capture the global structural features of plankton against occlusion and debris, we employ a Masked Autoencoder (MAE) for self-supervised pre-training on unlabeled individual images. Experimental results on real-world datasets demonstrate that our method significantly outperforms conventional methods, such as Mask R-CNN, particularly in challenging environments with high debris density. We demonstrate that our synthetic training strategy and MAE-based architecture enable high-precision segmentation with requiring less manual annotations for individual plankton images.

Masaharu Miyazaki, Yurie Otake, Koichi Ito +3

Graduate School of Information Sciences, Tohoku University 6-6-05, Aramaki Aza Aoba, Sendai, 9808579, Japan. · The Center for Ecological Research, Kyoto University, 2–509–3, Hirano, Otsu-shi, Shiga-ken, 5202113, Japan. · Graduate School of Life Sciences, Tohoku University, 6–3, Aramaki Aza Aoba, Aoba-ku, Sendai-shi, 9808578, Japan.

Instance Segmentation Synthetic Data Generation Microscopy Image Analysis

Ask PDF

Sep 10, 2026

Multimodal Taxonomic Conditioning for Generative Plankton Imagery

Automated plankton imaging produces severely long-tailed datasets, where the rare taxa of greatest ecological interest have too few images to train or evaluate classifiers reliably. We generate synthetic plankton imagery conditioned on taxonomy: a CLIP encoder is adapted on a large plankton corpus with a ranked contrastive objective extended to deep, ragged taxonomies, then frozen to condition a parameter-efficient diffusion transformer. We evaluate synthetic sample quality on distributional fidelity and downstream classifier utility.

Daniela Ivanova, Ozgu Goksu, Nicolas Pugeault

University of Glasgow, Glasgow, Scotland, UK · National Defence University, Istanbul, Türkiye

Contrastive Learning Synthetic Data Augmentation Conditional Image Generation Image Generation

Ask PDF

Abstract

Cross-Modal Representation Learning

Explore similar work

Planktonzilla: Multimodal dataset and models for understanding plankton ecosystems

PlankFormer: Robust Plankton Instance Segmentation via MAE-Pretrained Vision Transformers and Pseudo Community Image Generation

Multimodal Taxonomic Conditioning for Generative Plankton Imagery