A Mixed Diet Makes DINO An Omnivorous Vision Encoder

Rishabh Kabra; Maks Ovsjanikov; Drew A. Hudson; Ye Xia; Skanda Koppula; Andre Araujo; Joao Carreira; Niloy J. Mitra

cs.CVFeb 27, 2026

A Mixed Diet Makes DINO An Omnivorous Vision Encoder

Authors: Rishabh Kabra, Maks Ovsjanikov, Drew A. Hudson, Ye Xia, Skanda Koppula, Andre Araujo, Joao Carreira, Niloy J. Mitra

Organizations: Google DeepMind · University College London

Abstract

Pre-trained vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their features are poorly aligned across different visual modalities. For instance, the feature embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous Vision Encoder, a post-training framework that learns a modality-agnostic feature space. We fine-tune the encoder with a dual objective: first, to maximize the feature alignment between different modalities of the same scene; and second, a distillation objective that anchors the learned representations to a fully frozen teacher. The resulting student encoder becomes "omnivorous" by producing more consistent embeddings for a given scene, regardless of the input modality (RGB, Depth, Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model. Omnivorous model weights are available at https://github.com/google-deepmind/representations4d.

Figures & tables

Figure 1: Off-the-shelf vision encoders like DINO show poor cross-modal alignment. We show the similarity in feature space between randomly paired RGB images (top), between RGB images and depth maps of the same scene (middle), and between RGB and grayscale images of the same scene (bottom). While the numbers vary depending on the dataset, the pattern of misalignment between visual modalities remains consistent. Our proposed adapter aligns these modalities in an existing feature space.

Figure 2: Omnivorous Vision Encoder architecture. A frozen encoder

f^{*}

extracts features

z_{m}=f^{*}(x_{m})

from a spectrum of modalities denoted

m

(Segmentation, RGB, Depth). A trainable modality-agnostic adapter

g

maps these features into a common, aligned embedding space, producing a modality-invariant representation

h=g(z_{m})

. A convenient implementation of this architecture uses the early layers of a pretrained network as the frozen part

f^{*}

, and the later layers as the adapter

g

Figure 3: Training data: depth and segmentation maps are first colorized using a natural color palette derived from the corresponding RGB image. We then apply a data augmentation: we blend the colorized depth image with up to 50% of the RGB image (and likewise for the segmentation image). The compositing alpha is randomly sampled (between 0% to 50%) for each datapoint. The idea is to interpolate between the modalities (Depth $\leftrightarrow$ RGB $\leftrightarrow$ Seg) smoothly and teach the model a degree of invariance across the full spectrum, while also providing more negative examples for between-scene contrastive learning. Other potential benefits: the augmentation (i) makes our representations naturally invariant to scene lighting; and (ii) helps us cope with imperfect depth and segmentation values.

dataset	feature type	model	R@1 $\uparrow$	R@5 $\uparrow$	mAP $\uparrow$	MedR $\downarrow$
movi	gap	DINOv2 ViT-B/14	15.5	33.1	25.2	19.3
(N=128)	gap	Omnivorous ViT-B/14	86.2	96.5	90.9	1.0
	tok	DINOv2 ViT-B/14	18.2	34.5	27.2	16.8
	tok	Omnivorous ViT-B/14	76.6	92.7	83.4	1.0
scannet	gap	DINOv2 ViT-B/14	4.6	10.8	8.1	401.8
(N=3072)	gap	Omnivorous ViT-B/14	46.1	71.4	57.7	2.0

Table 1: Cross-modal retrieval: average results across all 6 directed modality pairs. We sample 1 frame per test video, yielding

N

queries and targets per dataset. GAP: Global Average Pooling of dense features. TOK: CLS Token embedding.

		depth delta1 $\uparrow$		depth rmse $\downarrow$		segmentation mean iou $\uparrow$
	dataset	navi probe3d	nyuv2	navi probe3d	nyuv2	ade20k	cityscapes	pascal voc
readout	model
Linear	DINOv2 ViT-B/14	0.697	0.875	0.076	0.405	0.463	0.622	0.814
Linear	Omnivorous ViT-B/14	0.706	0.896	0.074	0.377	0.475	0.632	0.826
DPT	DINOv2 ViT-B/14	0.779	0.948	0.061	0.297	0.496	0.737	0.855
DPT	Omnivorous ViT-B/14	0.781	0.948	0.061	0.297	0.505	0.732	0.857

Table 2: Downstream evals: monocular depth prediction and segmentation. We train either a Dense Prediction Transformer (DPT) or linear head on top of the frozen ViT backbone. Depth: The training minimizes a scale-invariant gradient loss and an edge-aware gradient loss. Evaluation is conducted on datasets like NYUv2 using standard metrics such as RMSE and threshold accuracy (

\delta_{i}=1.25^{i}

). Segmentation: The decoder heads are trained for pixel-wise classification. During evaluation, we compute Mean Intersection-over-Union (mIoU) by aggregating confusion matrices across batches.

feature type	model	accuracy $\uparrow$
tok	DINOv2 ViT-B/14	0.801
tok	Omnivorous ViT-B/14	0.835
tok & gap	DINOv2 ViT-B/14	0.804
	Omnivorous ViT-B/14	0.838

Table 3: Downstream eval: linear-probe classification on ImageNet. We sweep over five learning rates, picking the best one for each row. TOK: CLS Token embedding. TOK & GAP: both the CLS embedding and Average-Pooled dense features are used.

model	imagenet soft	inat	sop	gldv2	rp2k	food2k
DINOv2 ViT-B/14	81.936	78.53	54.39	51.90	66.83	51.90
Omnivorous ViT-B/14	81.974	77.49	54.69	50.13	70.48	52.14

Table 4: Downstream eval: k-NN classification . On ImageNet [ 8 ] , we follow the standard DINO evaluation protocol by using soft voting among the top-

k

neighbors (weighted by similarity) to predict classes, sweeping over multiple

k

values (e.g., 10, 20, 100) to report the best top-1 accuracy. On all other datasets (iNaturalist [ 16 ] , SOP [ 40 ] , GLDv2 [ 47 ] , RP2K [ 38 ] , Food2k [ 33 ] ), we use universal embeddings, evaluating the “hard” k-NN accuracy by matching test query embeddings against a training index of embeddings.

Panel of Table 5 — Table 5: Cross-modal transfer on the depth prediction task . Readout heads are trained on RGB images, but tested zero-shot on two novel modalities: Seg (within-distribution for Omnivorous) and NOCS images (out-of-distribution for both Omnivorous and DINO). The heads are trained on NYUv2, evaluated on PACE. We also show qualitative results for both models.

Panel of Figure 4 — Figure 4: Analysis of the anchoring loss. (a) Trade-off between cross-modal alignment and cross-scene discernibility, controlled by $\lambda_{anchor}$ . The x-axis measures alignment (cosine sim of $<$ RGB, Depth $>$ ) and the y-axis measures discernibility (1 - cosine similarity of distinct RGB scenes) on ScanNet. Frozen DINOv2 (light blue) is discriminative but poorly aligned. (b) To pick a value for $\lambda_{anchor}$ , we examine its effect on linear-head prediction performance, from Omnivorous features of RGB images, on Depth (NYUv2) and Segmentation (Cityscapes). We omit the datapoint for $\lambda_{anchor}=0$ located at $(x=0.732,y=0.356)$ for clarity, as it was too far below the remaining datapoints.

$\alpha_{max}$	classif. $\uparrow$	depth $\uparrow$	segment. $\uparrow$	3D corresp. $\uparrow$
0	0.831	0.899	0.624	28.40
0.25	0.834	0.898	0.630	28.96
0.5	0.834	0.896	0.632	29.00
0.75	0.834	0.894	0.632	29.04
1.0	0.835	0.891	0.632	29.03

Table 6: Ablating modality mixup. We vary

\alpha_{max}

which controls the degree of blending between modalities during training. We report linear-probe performance on (i) classification (ImageNet accuracy using the TOK feature, without intermediate layers), (ii) depth prediction (

\delta_{1}

on NYUv2), and (iii) segmentation (mean IoU on Cityscapes). We also report (iv) 3D correspondence (percentage of correct keypoints at threshold 0.0 on NAVI), which is assessed directly from features without a linear probe.

Panel of Figure 5 — Figure 5: PCA visualizations of frozen (DINO ViT-B/14) and adapted (Omnivorous ViT-B/14) features on two scenes.

Category	Details
Architecture	DINOv2 ViT-B/14 (173M parameters)
Architecture	Layers 0–7 are frozen, 8–11 are fine-tuned.
Optimizer	AdamW with learning rate $1\times 10^{-4}$
Compute	TPU v4 ( $4\times 4\times 4$ ) for 20,000 steps, with a total runtime of 1 hour 14 minutes
Batch Size	512 (Global)
Datasets	ScanNet [ 7 ] , TartanAir [ 46 ] , Hypersim [ 39 ] , MOVi [ 15 ] , PointOdyssey [ 53 ] , DynamicReplica [ 24 ]

Table 7: Training Configuration for Omnivorous DINO

	DINOv2 ViT-B/14				Omnivorous ViT-B/14
dataset	$<R,D>$	$<R,S>$	$<D,S>$	$<R_{1},R_{2}>$	$<R,D>$	$<R,S>$	$<D,S>$	$<R_{1},R_{2}>$
movi	0.263	0.284	0.481	0.237	0.567	0.579	0.721	0.279
scannet	0.285	0.216	0.413	0.198	0.600	0.550	0.663	0.259
tartanair	0.345	0.359	0.543	0.172	0.607	0.603	0.736	0.223

Table 8: Diagnostic metrics: we expand Fig 1, showing cross-modal alignment and cross-scene discernibility metrics across three datasets for both pretrained DINOv2 and the adapted Omnivorous model (at our default

\lambda_{anchor}=10)

). We denote the three modalities R, D, and S (RGB, Depth, and Segmentation, respectively). The metrics are computed without modality-mixup (i.e.,

\alpha_{max}=0

). For

<R_{1},R_{2}>

, lower similarity is considered better.

		absrel $\downarrow$	diff 11.25 $\uparrow$	diff 22.50 $\uparrow$	diff 30.00 $\uparrow$	mean diff angle $\downarrow$	rmse angle $\downarrow$
dataset	model
navi	DINOv2 ViT-B/14	197.9	43.5	72.2	82.1	18.6	24.6
navi	Omnivorous ViT-B/14	197.8	43.6	72.3	82.2	18.6	24.6
nyuv2	DINOv2 ViT-B/14	134.9	63.4	80.8	86.5	14.1	21.7
nyuv2	Omnivorous ViT-B/14	134.1	63.5	80.8	86.5	14.1	21.6

Table 9: Downstream eval: normals estimation using a DPT head.

block number	9	10	11	12
model
DINOv2 ViT-B/14	29.76	28.49	27.68	28.57
Omnivorous ViT-B/14	29.76	28.93	28.63	29.00

Table 10: Multiview correspondence: we report the Percentage of Correct Keypoints (

\uparrow

) at the 0.0 level (i.e., only exact matches are counted). We measure correspondence for all the four blocks that are fine-tuned in the Omnivorous case, comparing them with their frozen DINOv2 counterparts.

Panel of Figure 6 — Figure 6: Qualitative comparison (Omnivorous vs DINOv2) on depth prediction using a linear head. Please compare a versus b, and c versus d. We highlight notable differences using a black oval.

Panel of Figure 7 — Figure 7: Qualitative comparison (Omnivorous vs DINOv2) on segmentation prediction using a linear head. We highlight notable differences using a white oval.

Panel of Figure 8 — Figure 8: Qualitative comparison (Omnivorous vs DINOv2) on segmentation prediction (contd.) using a linear head. We highlight notable differences using a white oval.

Panel of Figure 9 — Figure 9: Behavior of Omnivorous TIPS.

	Classification (acc.)		Depth ( $\delta_{1}$ )		Segmentation (mean IoU)			Corresp. (PCK)
dataset	inet (linear)	inet (k-NN)	navi	nyuv2	ade20k	cityscapes	pascal voc	navi
parametrization
Adapter on top	0.840	81.832	0.679	0.905	0.470	0.628	0.826	28.15
Fine-tune final blocks	0.838	81.974	0.706	0.896	0.475	0.632	0.826	29.00

Table 11: Ablating the parametrization of the student: we either train a 4-block ViT on top of the DINOv2 ViT-B/14 backbone, or fine-tune the final 4 blocks of the backbone (ours). As before in Table 6 , we report metrics (all

\uparrow

) on (i) classification, using either linear probes on TOK & GAP, or k-NN, (ii) depth prediction (linear head), (iii) segmentation (linear head), and (iv) multiview correspondence.

		Depth ( $\delta_{1}$ )		Segmentation (mean IoU)
	dataset	navi	nyuv2	ade20k	cityscapes	pascal voc
readout	$L_{\text{stop-gradient}}$
DPT	4	0.777	0.948	0.495	0.727	0.855
	6	0.778	0.947	0.494	0.733	0.853
	8	0.781	0.948	0.505	0.732	0.857
	10	0.780	0.949	0.504	0.731	0.852

Table 12: Ablating the number of blocks kept frozen , denoted by

L_{\text{stop-gradient}}

, when training Omnivorous DINOv2. There are 12 total blocks in the ViT-B/14 architecture.

Cross-Modal Representation Learning

Cross-modal representation learning connects information across data types such as images, text, audio, biosignals, and sensor measurements.

See all 210 papers

Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder
Feb 27, 2026Kin Wai Lau, Yasar Abbas Ur Rehman, Lai-Man Po +1Multimodal Learning Unified Multimodal Models
City University of Hong Kong · TCL AI Lab · University of Surrey, United Kingdom
OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder
May 2, 2026Detao Bai, Shimin Yao, Weixuan Chen +4Cross-Modal Learning Video Understanding
1Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology
Invisible Shortcuts: Why Vision Encoders Know Your Camera
Aug 5, 2026Vladan Stojnić, Ryan Ramos, Giorgos Kordopatis-Zilos +2Vision Foundation Models Shortcut Learning
VRG, FEE, Czech Technical University in Prague · The University of Osaka

Feb 27, 2026J/K move · Enter open · S save

Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder

Kin Wai Lau, Yasar Abbas Ur Rehman, Lai-Man Po, Pedro Porto Buarque de Gusmão

City University of Hong Kong · TCL AI Lab · University of Surrey, United Kingdom

Multimodal Learning Unified Multimodal Models Multimodal Transformer Cross-Modal Representation Learning Efficient Multimodal Inference Multimodal Contrastive Learning

Open paper AskPDF arXiv

Recent multimodal systems often rely on separate expert modality encoders which cause linearly scaling complexity and computational overhead with added modalities. While unified Omni-models address this via Mixture-of-Expert (MoE) architectures with specialized experts and routing, they still inflate parameter counts and introduce routing overhead. In this paper, we propose Omni-C (Omni-Compress), a single dense Transformer-based encoder that learns competitive shared representations across heterogeneous modalities--images, audio, and text--through unimodal contrastive pretraining on large-scale unaligned data. By maximizing parameter sharing in the backbone and using lightweight modality-specific projection heads, Omni-C effectively mitigates inter-modality conflicts without requiring MoE, paired supervision, or routing. This design supports efficient deployment on memory-constrained systems via sequential modality processing and low-memory inference, eliminating the need for parallel expert loading or specialized hardware. Experiments show Omni-C achieves performance comparable to expert models in unimodal and cross-model tasks, with modest zero-shot degradation on audio and text that is largely recovered through lightweight linear probing or parameter efficient fine-tuning. The unified architecture substantially reduces inference memory usage compared to multi-encoder baselines, advancing efficient and scalable multimodal learning.

Abstract

Cross-Modal Representation Learning

Explore similar work

Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder

OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder

Invisible Shortcuts: Why Vision Encoders Know Your Camera