Conditional Neural Optimal Transport for Predicting Cellular Phenotypes from Molecular Structure

Gauthier Avité; Maxime Sanchez-Renauld; Nicolas Bourriez; Auguste Genovesio

cs.CVAug 14, 2026

Conditional Neural Optimal Transport for Predicting Cellular Phenotypes from Molecular Structure

Authors: Gauthier Avité, Maxime Sanchez-Renauld, Nicolas Bourriez, Auguste Genovesio

Organizations: IBENS, Ecole Normale Supérieure, Université PSL, Paris, France · Institut Curie, Université PSL, Paris, France · INSERM, Paris, France · Mines ParisTech, Université PSL, Paris, France ·

Abstract

High-content microscopy enables systematic profiling of cellular responses to chemical perturbations, but the scale of the chemical space makes exhaustive phenotypic characterization experimentally infeasible. This motivates computational models that can predict image-derived phenotypes without acquiring the corresponding treated cells. We formulate molecule-induced phenotype prediction as an inductive conditional transport problem in image representation space. Given a negative-control phenotype and the structure of a molecule, we aim to predict the phenotype induced by the corresponding molecule. We first evaluate classical optimal transport baselines and show that static couplings do not yield useful predictions on large-scale phenotypic image datasets. We then introduce a molecule-conditioned Neural Optimal Transport (NOT) model with a Monge-Gap regularization training objective that learns to transport negative-control unperturbed phenotypes toward perturbed phenotypes using molecular structure as conditioning information. NOT recovers molecule-specific phenotypic effects while reducing microscopy-associated technical variation, thereby facilitating comparisons across experimental batches. On unseen active molecules, the model outperforms baseline approaches, demonstrating that chemically conditioned transport can generalize beyond the molecules observed during training. We identified the molecular encoder as the main limitation to this generalization, while transport in a compressed representation space improves performance and scalability. These results establish NOT as a promising framework for predicting cellular phenotypes from molecular structure and negative-control phenotypes, while highlighting the development of more informative molecular representations as a key direction for improving out-of-distribution performance.

Figures & tables

Figure 1: Overview of the molecule-conditioned Neural Optimal Transport framework. (a) Model architecture . DMSO images and molecular structures are encoded separately and projected to a common dimension $d$ , then combined through cross-attention to predict perturbation-induced phenotype embeddings. (b) Training objective . The conditional map $T_{\theta}(\cdot\mid f)$ transforms the source distribution $\mu$ into $T_{\theta\#}\mu$ . Learning minimizes a fitting loss between the predicted and target distributions, together with a Monge-Gap regularizer that constrains the transport. (c) Evaluation . Predictions are ranked against candidate target phenotypes by cosine similarity. Retrieval is successful when the correct perturbation appears among the top $k$ candidates, using held-out plates for in-distribution evaluation and held-out molecules for out-of-distribution evaluation.

Panel of Figure 2 — Figure 2: In-distribution prediction on held-out plates. (a) Retrieval@1 for the eight shared positive controls, reported as mean $\pm$ standard deviation across five folds and three seeds. (b) UMAP of source, target, and predicted phenotypes for the four most active controls. The projection was fitted jointly on source and target embeddings.

Figure 3: Generated phenotypes improve cross-plate comparability. Cross-plate retrieval performance on held-out plates. Bars show the mean mAP across controls, while points denote individual molecules. Real phenotypes are evaluated either directly or after per-plate DMSO-based normalization. Generated phenotypes achieve higher cross-plate mAP than both real baselines.

Panel of Figure 4 — Figure 4: Neural OT on the molecule-level OOD split (5-fold, 3-seed mean $\pm$ std). (a) Replicate- and plate-level Retrieval@10 for NOT (ours) and the identity mapping: the conditioned map clearly outperforms the baseline. (b) The same metrics for four molecular structure encoders under an otherwise identical NOT, ordered by plate-level retrieval: retrieval improves substantially with encoder quality. Bars show the mean and error bars the standard deviation over five folds and three seeds. The dashed line marks the chance level ( $k/M=10/2248$ ).

Table 1: Impact of dimensionality reduction on OOD retrieval (5-fold, 3-seed mean

\pm

std; best in bold, second underlined). (a) Reducing the conditioning embedding sharply hurts retrieval. (b) Reducing the phenotype embedding helps, the 50D autoencoder being best; this ablation was run at

\tau{=}0.95

, the near-optimal unbalancedness of Table 10 (Appendix 0.C.4 ).

Table 2: Ablations on the Monge-Gap strength and the conditioning mechanism (5-fold, 3-seed mean

\pm

std; best in bold, second underlined). (a) Monge-Gap strength

\lambda

: the geometry-preserving penalty trades replicate for plate retrieval. (b) Injecting the molecular structure by multi-head attention versus plain concatenation.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Examples of images from the eight positive controls of JUMP-CP.

Figure 6: Distribution of molecule-level phenotypic distances to DMSO negative controls. The red vertical line indicates the threshold separating the top ( $10\%$ ) most phenotypically active molecules, retained for the out-of-distribution evaluation, from the remaining molecules.

Element	Value
Seeds	1, 2, 3
Split strategy	Plate-level 5-fold cross-validation
Train/validation split	80/20 per fold
Number of train/validation plates	$\approx$ 1285 train, $\approx$ 322 validation per fold
Image embedding reduction	Autoencoder, 50 dimensions
Molecular condition	Morgan fingerprint

Appendix

Table 3: Experimental setup for NOT training.

Element	Value
Architecture	Attention conditional network
Phenotypic input	AE-reduced DINOv2 embedding, 50 dimensions
Conditional input	Morgan fingerprint, 1024 bits
Fingerprint encoder	MLP: $1024\rightarrow 512\rightarrow 512\rightarrow 50$ , ReLU activations
Number of attention heads	2
Feed-forward hidden dimension	64

Appendix

Table 4: Neural OT model architecture.

Hyperparameter	Value
Number of epochs	50
Optimizer	Adam
Learning rate	$10^{-4}$
Learning-rate scheduler	Warmup + constant + cosine decay
Warmup fraction	0.2
Warmup epochs	10

Appendix

Table 5: Optimization hyperparameters.

Element	Value
Training objective	Sinkhorn fitting loss + Monge-gap regularization
Cost type	Cosine cost
$\varepsilon_{\mathrm{fit}}$	0.01
Sinkhorn fitting blur	$\sqrt{0.01}=0.1$
Fitting debias	True
$\varepsilon_{\mathrm{reg}}$	0.1

Appendix

Table 6: Optimal transport loss configuration.

Element	Value
Split strategy	Molecule-level OOD (disjoint train/test molecules)
Cross-validation	5-fold $\times$ 3 seeds
molecule subset	Most-active molecules
Phenotype embedding	DINOv2-Giant, autoencoder-reduced to 50D
Structure encoder (baseline)	morganc+rdkc , radius 3; no reduction
Architecture	Attention conditional network; EMA decay 0.99

Appendix

Table 7: Hyperparameters for the molecule-level OOD ablation (Section 5.2 ). The settings that differ from the in-distribution setup of Appendix 0.C.2 are the data split, the entropic scales

\varepsilon_{\mathrm{fit}},\varepsilon_{\mathrm{reg}}

, the Monge-gap weight

\lambda

, the number of epochs, and the structure encoder.

Structure encoder	R@10 rep	R@10 plate
morganc+rdkc	0.035 $\pm$ 0.002	0.095 $\pm$ 0.007
morgan	0.028 $\pm$ 0.001	0.071 $\pm$ 0.007
molformer	0.017 $\pm$ 0.001	0.040 $\pm$ 0.003
unimol2	0.015 $\pm$ 0.001	0.029 $\pm$ 0.003

Appendix

Table 8: Structure encoders under NOT on the molecule-level OOD split (5-fold, 3-seed mean

\pm

std). Best in bold, second best underlined. These are the values plotted in Figure 4(b) .

Objective / cost	R@10 rep	R@10 plate
NOT (cosine cost)	0.035 $\pm$ 0.002	0.095 $\pm$ 0.007
NOT ( $\ell_{2}$ cost)	0.038 $\pm$ 0.002	0.095 $\pm$ 0.006

Appendix

Table 9: Ablation on the training objective and ground cost (5-fold, 3-seed mean

\pm

std). Best in bold, second best underlined.

$\tau$ (unbalanced resampler)	R@10 rep	R@10 plate
1.00	0.035 $\pm$ 0.002	0.095 $\pm$ 0.007
0.95	0.037 $\pm$ 0.002	0.101 $\pm$ 0.007
0.90	0.037 $\pm$ 0.001	0.101 $\pm$ 0.007

Appendix

Table 10: Unbalancedness

\tau

of the resampling heuristic (5-fold, 3-seed mean

\pm

std). Best in bold, second best underlined. Mild unbalancedness (

\tau\approx 0.90

–

0.95

) gives small, consistent plate gains over the balanced

\tau{=}1

baseline.

Method	Phenotype distance	# correct (out of 1125)
GW	$\ell_{2}$	0
GW	cosine	1
Random coupling	—	1
Semi-supervised FGW ( $80\%$ labels, $\lambda{=}100$ )	cosine	3

Appendix

Table 11: Static GW/FGW on the most-active JUMP-CP subset (

1125

molecules): number of correct structure

\to

phenotype matches. Even

80\%

of supervised correspondences leaves the coupling at chance level.

Optimal Transport

Optimal transport compares and transforms probability distributions by finding low-cost ways to move mass between them.

See all 291 papers

Learning Molecular Representations from Cellular Phenotypes with Structure Preservation
Aug 3, 2026Xuan Lin, Jingyu Sheng, Tengfei Ma +2Disentangled Representation Learning Cross-Modal Representation Learning
School of Computer Science, Xiangtan University, Xiangtan, Hunan, China · College of Computer Science and Electronic Engineering, Hunan University, Changsha, Hunan, China · School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China +1
Can phenotypic activity be predicted without experimental readouts?
Oct 6, 2026Télio Cropsal, Rocío MercadoMolecular Property Prediction Molecular Representation Learning
AI Laboratory for Molecular Engineering (AIME) Department of Computer Science and Engineering Chalmers University of Technology & University of Gothenburg, Gothenburg, Sweden · AI Laboratory for Molecular Engineering (AIME) Department of Computer Science and Engineering Chalmers University of Technology & University of Gothenburg and Science for Life Laboratory (SciLifeLab), Gothenburg, Sweden
Intervention-Aware Multiscale Representation Learning from Imaging Phenomics and Perturbation Transcriptomics
Apr 19, 2026Jiayuan Chen, Ruoqi Liu, Zishan Gu +1Cross-Modal Knowledge Distillation Cross-Modal Representation Learning
The Ohio State University

Aug 3, 2026J/K move · Enter open · S save

Learning Molecular Representations from Cellular Phenotypes with Structure Preservation

Xuan Lin, Jingyu Sheng, Tengfei Ma, Li Sun+1

School of Computer Science, Xiangtan University, Xiangtan, Hunan, China · College of Computer Science and Electronic Engineering, Hunan University, Changsha, Hunan, China · School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China+1

Disentangled Representation Learning Cross-Modal Representation Learning Molecular Property Prediction Molecular Representation Learning Drug Discovery

Open paper AskPDF arXiv

Phenotypic drug discovery enables the discovery of functional relationships between molecular structures and cellular responses. However, existing multimodal representation learning methods often optimize cross-modal alignment without considering the intrinsic organization of chemical space, resulting in distorted molecular representations and loss of structural information. We propose PhenMol, a structure-preserving framework for phenotype-aware molecular representation learning. PhenMol disentangles molecular and cellular representations into shared and private components, enabling phenotype-guided alignment while preserving chemical structures through a dedicated molecular branch. This design integrates cellular phenotype information without disrupting molecular neighborhood organization. Experiments on approximately

3.04 \times 10^{4}

molecule--cell morphology pairs demonstrate that PhenMol improves molecular property prediction across 270 bioactivity tasks, molecule--phenotype retrieval, and clinical trial outcome prediction. Moreover, ECFP4-based structural analysis shows that PhenMol better preserves molecular neighborhoods and reduces embedding distortion compared with existing multimodal alignment methods. These results highlight the importance of structure-aware constraints in multimodal molecular representation learning and provide an effective approach for integrating cellular phenotypes with chemical knowledge for drug discovery.

Abstract

Optimal Transport

Explore similar work

Learning Molecular Representations from Cellular Phenotypes with Structure Preservation

Can phenotypic activity be predicted without experimental readouts?

Intervention-Aware Multiscale Representation Learning from Imaging Phenomics and Perturbation Transcriptomics