Probing Intrinsic Medical Task Relationships: A Contrastive Learning Perspective

Jonas Muth; Zdravko Marinov; Simon Reiß

cs.CVApr 7, 2026

Probing Intrinsic Medical Task Relationships: A Contrastive Learning Perspective

Authors: Jonas Muth, Zdravko Marinov, Simon Reiß

Organizations: Karlsruhe Institute of Technology, Karlsruhe, Germany

Abstract

While much of the medical computer vision community has focused on advancing performance for specific tasks, the underlying relationships between tasks, i.e., how they relate, overlap, or differ on a representational level, remain largely unexplored. Our work explores these intrinsic relationships between medical vision tasks, specifically, we investigate 30 tasks, such as semantic tasks (e.g., segmentation and detection), image generative tasks (e.g., denoising, inpainting, or colorization), and image transformation tasks (e.g., geometric transformations). Our goal is to probe whether a data-driven representation space can capture an underlying structure of tasks across a variety of 39 datasets from wildly different medical imaging modalities, including computed tomography, magnetic resonance, electron microscopy, X-ray ultrasound and more. By revealing how tasks relate to one another, we aim to provide insights into their fundamental properties and interconnectedness. To this end, we introduce Task-Contrastive Learning (TaCo), a contrastive learning framework designed to embed tasks into a shared representation space. Through TaCo, we map these heterogeneous tasks from different modalities into a joint space and analyze their properties: identifying which tasks are distinctly represented, which blend together, and how iterative alterations to tasks are reflected in the embedding space. Our work provides a foundation for understanding the intrinsic structure of medical vision tasks, offering a deeper understanding of task similarities and their interconnected properties in embedding spaces.

Figures & tables

Figure 1: We aim to train a model, which is able to embed medical computer vision tasks into a joint representation space. This space allows us to explore how tasks relate to one another: for example, we can analyze the proximity of unseen tasks to known tasks (right) or study how iterative task alterations ( e.g. , increasing noise) are reflected in the learned embedding space (left).

Figure 2: Overview of our Task-Contrastive Learning . We sample input-output image pairs from visual tasks and embed them with a neural network into vectors, which is tuned with a contrastive loss.

Classification into: visual tasks	seen tasks			unseen tasks			all tasks
	$k=1$	$k=3$	$k=5$	$k=1$	$k=3$	$k=5$	$k=1$	$k=3$	$k=5$
	Nearest neighbor classification on seen datasets
CLIP ( 17 )	$73.6\%$	$73.8\%$	$74.0\%$	$82.4\%$	$82.2\%$	$81.7\%$	$70.9\%$	$71.1\%$	$71.1\%$
ImageNet ( 34 )	$71.7\%$	$71.9\%$	$71.8\%$	$81.1\%$	$81.0\%$	$80.7\%$	$69.1\%$	$69.2\%$	$69.1\%$
SimCLR ( 15 )	$75.1\%$	$74.7\%$	$73.9\%$	$84.0\%$	$83.1\%$	$82.1\%$	$70.6\%$	$70.2\%$	$69.6\%$
TaCo (ours)	$\mathbf{85.2}\%$	$\mathbf{85.0}\%$	$\mathbf{84.6}\%$	$\mathbf{85.9}\%$	$\mathbf{85.4}\%$	$\mathbf{84.6}\%$	$\mathbf{78.4}\%$	$\mathbf{78.7}\%$	$\mathbf{78.3}\%$

Table 1: Results for kNN classification of vision tasks in F1-scores. We evaluate a ResNet pre-trained on ImageNet, the multi-modal CLIP vision encoder, the contrastive baseline SimCLR trained in-domain, and our method TaCo trained without and with failure tasks .

Seen tasks on seen datasets
Classification into:	tasks	datasets
CLIP ( 17 )	$85.1\%$	$85.6\%$
ImageNet ( 34 )	$80.9\%$	$86.0\%$
SimCLR ( 15 )	$82.7\%$	$86.2\%$
TaCo (ours)	$\textbf{96.4}\%$	$\textbf{89.4}\%$
TaCo (ours) + failure tasks	$95.3\%$	$84.4\%$

Table 2: Nearest neighbor classification (

k=5

) in F1-Score of task and dataset classification of different representations. Insight into method’s focus on tasks or datasets.

	Seen tasks and seen datasets			Unseen tasks and unseen datasets			Seen and unseen tasks and datasets
Classification into:	tasks	datasets	visual tasks	tasks	datasets	visual tasks	tasks	datasets	visual tasks
SimCLR	$82.7\%$	$86.2\%$	$73.9\%$	$93.2\%$	$\mathbf{99.6}\%$	$93.5\%$	$78.2\%$	$82.5\%$	$67.0\%$
TaCo	$89.8\%$	$80.9\%$	$71.9\%$	$93.8\%$	$98.2\%$	$92.0\%$	$83.8\%$	$77.1\%$	$63.5\%$
+ balanced task sampling	$\mathbf{96.4}\%$	$\mathbf{89.4}\%$	$\mathbf{84.6}\%$	$\mathbf{96.9}\%$	$98.8\%$	$95.0\%$	$89.2\%$	$\mathbf{84.1}\%$	$\mathbf{74.0}\%$
+ failure tasks	$95.3\%$	$84.4\%$	$80.3\%$	$96.4\%$	$97.9\%$	$\mathbf{96.1}\%$	$\mathbf{90.2}\%$	$78.7\%$	$70.6\%$

Table 3: Ablation for kNN (

k=5

) classification of tasks, datasets and visual tasks for seen, unseen and all visual tasks in F-1 Scores.

Panel of Figure 3 — Figure 3: Examples for iterative task adaptation. Top : Brightness is changed successively. Middle : Image is successively rotated. Bottom : Segmentation of an image is successively noised. Only subsequences are displayed here for brevity.

Panel of Figure 4 — Figure 4: Iterative task adaptation. First row: Distances in CLIP representation space, second row: distances in TaCo representation space. Third row: distances in TaCo + failure tasks representation space. Columns indicate iterative task adaptations for the tasks: changing brightness, rotating an image and adding successively more noise to a segmentation task. The x-axes describe the magnitude of task adaptation, colored areas relate to standard deviation. Note: The x-axis in the noisy segmentation plots first has a step size of $2\%$ and successively a step size of $10\%$ .

Panel of Figure 5 — Figure 5: Learned high dimensional task representation spaces projected via t-SNE. We showcase the embedding space of CLIP, TaCo and TaCo + failure tasks . Colors indicate different tasks (top row) and different datasets (bottom row), best viewed zooming in.

Panel of Figure 6 — Figure 6: Task similarity matrices based on the cosine similarities between task representations of different base tasks. The right matrix is row-wise normalied.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

AMOSMR	autoPET	BraTS T1	BraTS T1CE	CDD-CESM
Cervical Cancer	Chest-Xray-M.-L.	COVID-CT-Seg.	COVID-QU-Ex-inf.	COVID-QU-Ex-L.
COVID19-CT-Segb.	crossmoda	hc18	Heart	Intraretinal-Cyst.
ISLES2022 ADC	ISLES2022 DWI	Kvasir-SEG	LungLesions	m2caiSeg
NeurIPS22CellSeg	OpenOrganelle hela-2	OpenOrganelle hela-3	OpenOrg. jurkat-1	OpenOrg. marco-2
OpenOrg. sum159	Pneumothorax M.	ProstateADC	ProstateT2	QIN-Prostate-L.
QIN-Prostate-P.	SpineMR	TCIA-LCTSC	WMH flair	WMH T1

Appendix

Table 4: Example images and segmentations of the datasets from MedSAM Ma et al. [2024a] and OpenOrganelle Heinrich et al. [2021] . Here, we highlight unseen datasets through yellow tinted tiles.

Dataset	Link	Licence
AMOSMR Ji et al. [2022]	https://amos22.grand-challenge.org/Dataset/	CC BY-NC-SA
autoPET Gatidis et al. [2022]	https://www.cancerimagingarchive.net/collection/fdg-pet-ct-lesions/	TCIA Restricted
BraTS T1 Bakas et al. [2017a] , Bakas et al. [2017b] , Bakas et al. [2017c] , Bakas et al. [2018] , Menze et al. [2014]	http://braintumorsegmentation.org/	Data Usage Agreement
BraTS T1CE Bakas et al. [2017a] , Bakas et al. [2017b] , Bakas et al. [2017c] , Bakas et al. [2018] , Menze et al. [2014]	http://braintumorsegmentation.org/	Data Usage Agreement
CDD-CESM Khaled and others [2021]	https://www.cancerimagingarchive.net/collection/cdd-cesm/	CC BY 4.0
Cervical Cancer Mayr et al. [2023]	https://www.cancerimagingarchive.net/collection/cc-tumor-heterogeneity/	CC BY 4.0

Appendix

Table 5: Datasets with their citation, link to the respective website and license.

Panel of Table 6 — Table 6: Semantic tasks from our datasets. Here, we highlight unseen tasks via yellow tinted tiles.

Panel of Table 7 — Table 7: Image transformation tasks from our datasets. Yellow tinted tiles: unseen tasks.

Panel of Table 8 — Table 8: Image generative tasks from our datasets. Yellow tinted tiles: unseen tasks.

	SimCLR	TaCo	TaCo + failure tasks
epochs	3	3	3
batch size	$2\cdot 20=\textbf{40}$	$2\cdot 20=\textbf{40}$	$15\cdot 2+10=\textbf{40}$
learning rate	$0.001$	$0.001$	$0.001$
failure tasks start iteration	–	–	$50.000$
$\#$ failure tasks	–	–	10
weight decay	$0.0001$	$0.0001$	$0.0001$

Appendix

Table 9: Hyperparameters for the training of SimCLR Chen et al. [2020a] and our TaCo models.

Panel of Figure 7 — Figure 7: Learned high dimensional representations from the seen tasks and seen datasets scenario projected via t-SNE. We showcase the embedding space of CLIP, TaCo and TaCo + failure tasks . Coloring is based on tasks in the first row and datasets in the second row.

Panel of Figure 8 — Figure 8: Learned high dimensional representations from the unseen tasks and unseen datasets scenario projected via t-SNE. We showcase the embedding space of CLIP, TaCo and TaCo + failure tasks . Coloring is based on tasks in the first row and datasets in the second row.

Explore similar work

MultiMedVision: Multi-Modal Medical Vision Framework
May 9, 2026Frank Li, Bardia Khosravi, Mohammadreza Chavoshi +5Medical Imaging Foundation Models Vision Transformer
Emory University · Yale University
Med-DisSeg: Dispersion-Driven Representation Learning for Fine-Grained Medical Image Segmentation
May 14, 2026Zhiquan Chen, Haitao Wang, Guowei Zou +1Image Segmentation Metric Learning
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, China
Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelines
May 18, 2026Chathura Wimalasiri, Kishor Nandakishor, Marimuthu PalaniswamiContrastive Learning Self-Supervised Learning
Department of Electrical and Electronic Engineering, University of Melbourne, Parkville, VIC 3052, Australia

May 9, 2026J/K move · Enter open · S save

MultiMedVision: Multi-Modal Medical Vision Framework

Frank Li, Bardia Khosravi, Mohammadreza Chavoshi, Young Seok Jeon+4

Emory University · Yale University

Medical Imaging Foundation Models Vision Transformer Multimodal Pretraining Multimodal Foundation Models Medical Image Representation Learning

Open paper AskPDF arXiv

Multi-modal medical imaging enables comprehensive diagnostics, yet current foundation models process 2D (e.g. X-ray) and 3D (e.g. CT) data with separate, dimensionality-specific architectures. We present MultiMedVision, a unified framework for joint 2D/3D representation learning built on a Sparse Vision Transformer. Our model uses 3D Rotary Positional Embeddings and variable-length sequence packing to process mixed-modality batches natively within a shared latent space, without modality-specific adapters or treating 3D volumes as 2D slice sequences. Trained with a self-supervised objective on chest X-rays (MIMIC-CXR) and CT scans (CT-RATE), and using a single shared encoder with 5x less data, MultiMedVision achieves competitive performance on both 2D benchmarks (Macro AUROC 0.82 on MIMIC, 0.84 on CheXpert) and 3D tasks (0.85 on CT-RATE). Analysis of the learned representations reveals coexisting modality-specific and shared feature subspaces, demonstrating that unified cross-dimensional representation learning is feasible without sacrificing modality-specific performance.

Abstract

Explore similar work

MultiMedVision: Multi-Modal Medical Vision Framework

Med-DisSeg: Dispersion-Driven Representation Learning for Fine-Grained Medical Image Segmentation

Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelines