Pre-trained vision encoders like DINOv2 have demonstrated exceptional performance on unimodal tasks. However, we observe that their features are poorly aligned across different visual modalities. For instance, the feature embedding for an RGB image and its corresponding depth map of the same scene exhibit a cosine similarity that is nearly identical to that of two random, unrelated images. To address this, we propose the Omnivorous Vision Encoder, a post-training framework that learns a modality-agnostic feature space. We fine-tune the encoder with a dual objective: first, to maximize the feature alignment between different modalities of the same scene; and second, a distillation objective that anchors the learned representations to a fully frozen teacher. The resulting student encoder becomes "omnivorous" by producing more consistent embeddings for a given scene, regardless of the input modality (RGB, Depth, Segmentation, etc.). This approach enables robust cross-modal understanding while retaining the discriminative semantics of the original foundation model. Omnivorous model weights are available at https://github.com/google-deepmind/representations4d.
Figures & tables
Figure 1: Off-the-shelf vision encoders like DINO show poor cross-modal alignment. We show the similarity in feature space between randomly paired RGB images (top), between RGB images and depth maps of the same scene (middle), and between RGB and grayscale images of the same scene (bottom). While the numbers vary depending on the dataset, the pattern of misalignment between visual modalities remains consistent. Our proposed adapter aligns these modalities in an existing feature space.
Figure 2: Omnivorous Vision Encoder architecture. A frozen encoder f∗ extracts features zm=f∗(xm) from a spectrum of modalities denoted m (Segmentation, RGB, Depth). A trainable modality-agnostic adapter g maps these features into a common, aligned embedding space, producing a modality-invariant representation h=g(zm) . A convenient implementation of this architecture uses the early layers of a pretrained network as the frozen part f∗ , and the later layers as the adapter g .
Figure 3: Training data: depth and segmentation maps are first colorized using a natural color palette derived from the corresponding RGB image. We then apply a data augmentation: we blend the colorized depth image with up to 50% of the RGB image (and likewise for the segmentation image). The compositing alpha is randomly sampled (between 0% to 50%) for each datapoint. The idea is to interpolate between the modalities (Depth ↔ RGB ↔ Seg) smoothly and teach the model a degree of invariance across the full spectrum, while also providing more negative examples for between-scene contrastive learning. Other potential benefits: the augmentation (i) makes our representations naturally invariant to scene lighting; and (ii) helps us cope with imperfect depth and segmentation values.
dataset
feature type
model
R@1 ↑
R@5 ↑
mAP ↑
MedR ↓
movi
gap
DINOv2 ViT-B/14
15.5
33.1
25.2
19.3
(N=128)
Omnivorous ViT-B/14
86.2
96.5
90.9
1.0
tok
DINOv2 ViT-B/14
18.2
34.5
27.2
16.8
Omnivorous ViT-B/14
76.6
92.7
83.4
1.0
scannet
gap
DINOv2 ViT-B/14
4.6
10.8
8.1
401.8
(N=3072)
Omnivorous ViT-B/14
46.1
71.4
57.7
2.0
Table 1: Cross-modal retrieval: average results across all 6 directed modality pairs. We sample 1 frame per test video, yielding N queries and targets per dataset. GAP: Global Average Pooling of dense features. TOK: CLS Token embedding.
depth delta1 ↑
depth rmse ↓
segmentation mean iou ↑
dataset
navi probe3d
nyuv2
navi probe3d
nyuv2
ade20k
cityscapes
pascal voc
readout
model
Linear
DINOv2 ViT-B/14
0.697
0.875
0.076
0.405
0.463
0.622
0.814
Omnivorous ViT-B/14
0.706
0.896
0.074
0.377
0.475
0.632
0.826
DPT
DINOv2 ViT-B/14
0.779
0.948
0.061
0.297
0.496
0.737
0.855
Omnivorous ViT-B/14
0.781
0.948
0.061
0.297
0.505
0.732
0.857
Table 2: Downstream evals: monocular depth prediction and segmentation. We train either a Dense Prediction Transformer (DPT) or linear head on top of the frozen ViT backbone. Depth: The training minimizes a scale-invariant gradient loss and an edge-aware gradient loss. Evaluation is conducted on datasets like NYUv2 using standard metrics such as RMSE and threshold accuracy ( δi=1.25i ). Segmentation: The decoder heads are trained for pixel-wise classification. During evaluation, we compute Mean Intersection-over-Union (mIoU) by aggregating confusion matrices across batches.
feature type
model
accuracy ↑
tok
DINOv2 ViT-B/14
0.801
Omnivorous ViT-B/14
0.835
tok & gap
DINOv2 ViT-B/14
0.804
Omnivorous ViT-B/14
0.838
Table 3: Downstream eval: linear-probe classification on ImageNet. We sweep over five learning rates, picking the best one for each row. TOK: CLS Token embedding. TOK & GAP: both the CLS embedding and Average-Pooled dense features are used.
model
imagenet soft
inat
sop
gldv2
rp2k
food2k
DINOv2 ViT-B/14
81.936
78.53
54.39
51.90
66.83
51.90
Omnivorous ViT-B/14
81.974
77.49
54.69
50.13
70.48
52.14
Table 4: Downstream eval: k-NN classification . On ImageNet [ 8 ] , we follow the standard DINO evaluation protocol by using soft voting among the top- k neighbors (weighted by similarity) to predict classes, sweeping over multiple k values (e.g., 10, 20, 100) to report the best top-1 accuracy. On all other datasets (iNaturalist [ 16 ] , SOP [ 40 ] , GLDv2 [ 47 ] , RP2K [ 38 ] , Food2k [ 33 ] ), we use universal embeddings, evaluating the “hard” k-NN accuracy by matching test query embeddings against a training index of embeddings.
Table 5: Cross-modal transfer on the depth prediction task . Readout heads are trained on RGB images, but tested zero-shot on two novel modalities: Seg (within-distribution for Omnivorous) and NOCS images (out-of-distribution for both Omnivorous and DINO). The heads are trained on NYUv2, evaluated on PACE. We also show qualitative results for both models.
Figure 4: Analysis of the anchoring loss. (a) Trade-off between cross-modal alignment and cross-scene discernibility, controlled by λanchor . The x-axis measures alignment (cosine sim of < RGB, Depth > ) and the y-axis measures discernibility (1 - cosine similarity of distinct RGB scenes) on ScanNet. Frozen DINOv2 (light blue) is discriminative but poorly aligned. (b) To pick a value for λanchor , we examine its effect on linear-head prediction performance, from Omnivorous features of RGB images, on Depth (NYUv2) and Segmentation (Cityscapes). We omit the datapoint for λanchor=0 located at (x=0.732,y=0.356) for clarity, as it was too far below the remaining datapoints.
αmax
classif. ↑
depth ↑
segment. ↑
3D corresp. ↑
0
0.831
0.899
0.624
28.40
0.25
0.834
0.898
0.630
28.96
0.5
0.834
0.896
0.632
29.00
0.75
0.834
0.894
0.632
29.04
1.0
0.835
0.891
0.632
29.03
Table 6: Ablating modality mixup. We vary αmax which controls the degree of blending between modalities during training. We report linear-probe performance on (i) classification (ImageNet accuracy using the TOK feature, without intermediate layers), (ii) depth prediction ( δ1 on NYUv2), and (iii) segmentation (mean IoU on Cityscapes). We also report (iv) 3D correspondence (percentage of correct keypoints at threshold 0.0 on NAVI), which is assessed directly from features without a linear probe.
Figure 5: PCA visualizations of frozen (DINO ViT-B/14) and adapted (Omnivorous ViT-B/14) features on two scenes.
Category
Details
Architecture
DINOv2 ViT-B/14 (173M parameters)
Layers 0–7 are frozen, 8–11 are fine-tuned.
Optimizer
AdamW with learning rate 1×10−4
Compute
TPU v4 ( 4×4×4 ) for 20,000 steps, with a total runtime of 1 hour 14 minutes
Table 7: Training Configuration for Omnivorous DINO
DINOv2 ViT-B/14
Omnivorous ViT-B/14
dataset
<R,D>
<R,S>
<D,S>
<R1,R2>
<R,D>
<R,S>
<D,S>
<R1,R2>
movi
0.263
0.284
0.481
0.237
0.567
0.579
0.721
0.279
scannet
0.285
0.216
0.413
0.198
0.600
0.550
0.663
0.259
tartanair
0.345
0.359
0.543
0.172
0.607
0.603
0.736
0.223
Table 8: Diagnostic metrics: we expand Fig 1, showing cross-modal alignment and cross-scene discernibility metrics across three datasets for both pretrained DINOv2 and the adapted Omnivorous model (at our default λanchor=10) ). We denote the three modalities R, D, and S (RGB, Depth, and Segmentation, respectively). The metrics are computed without modality-mixup (i.e., αmax=0 ). For <R1,R2> , lower similarity is considered better.
absrel ↓
diff 11.25 ↑
diff 22.50 ↑
diff 30.00 ↑
mean diff angle ↓
rmse angle ↓
dataset
model
navi
DINOv2 ViT-B/14
197.9
43.5
72.2
82.1
18.6
24.6
Omnivorous ViT-B/14
197.8
43.6
72.3
82.2
18.6
24.6
nyuv2
DINOv2 ViT-B/14
134.9
63.4
80.8
86.5
14.1
21.7
Omnivorous ViT-B/14
134.1
63.5
80.8
86.5
14.1
21.6
Table 9: Downstream eval: normals estimation using a DPT head.
block number
9
10
11
12
model
DINOv2 ViT-B/14
29.76
28.49
27.68
28.57
Omnivorous ViT-B/14
29.76
28.93
28.63
29.00
Table 10: Multiview correspondence: we report the Percentage of Correct Keypoints ( ↑ ) at the 0.0 level (i.e., only exact matches are counted). We measure correspondence for all the four blocks that are fine-tuned in the Omnivorous case, comparing them with their frozen DINOv2 counterparts.
Figure 6: Qualitative comparison (Omnivorous vs DINOv2) on depth prediction using a linear head. Please compare a versus b, and c versus d. We highlight notable differences using a black oval.
Figure 7: Qualitative comparison (Omnivorous vs DINOv2) on segmentation prediction using a linear head. We highlight notable differences using a white oval.
Figure 8: Qualitative comparison (Omnivorous vs DINOv2) on segmentation prediction (contd.) using a linear head. We highlight notable differences using a white oval.
Figure 9: Behavior of Omnivorous TIPS.
Classification (acc.)
Depth ( δ1 )
Segmentation (mean IoU)
Corresp. (PCK)
dataset
inet (linear)
inet (k-NN)
navi
nyuv2
ade20k
cityscapes
pascal voc
navi
parametrization
Adapter on top
0.840
81.832
0.679
0.905
0.470
0.628
0.826
28.15
Fine-tune final blocks
0.838
81.974
0.706
0.896
0.475
0.632
0.826
29.00
Table 11: Ablating the parametrization of the student: we either train a 4-block ViT on top of the DINOv2 ViT-B/14 backbone, or fine-tune the final 4 blocks of the backbone (ours). As before in Table 6 , we report metrics (all ↑ ) on (i) classification, using either linear probes on TOK & GAP, or k-NN, (ii) depth prediction (linear head), (iii) segmentation (linear head), and (iv) multiview correspondence.
Depth ( δ1 )
Segmentation (mean IoU)
dataset
navi
nyuv2
ade20k
cityscapes
pascal voc
readout
Lstop-gradient
DPT
4
0.777
0.948
0.495
0.727
0.855
6
0.778
0.947
0.494
0.733
0.853
8
0.781
0.948
0.505
0.732
0.857
10
0.780
0.949
0.504
0.731
0.852
Table 12: Ablating the number of blocks kept frozen , denoted by Lstop-gradient , when training Omnivorous DINOv2. There are 12 total blocks in the ViT-B/14 architecture.
Recent multimodal systems often rely on separate expert modality encoders which cause linearly scaling complexity and computational overhead with added modalities. While unified Omni-models address this via Mixture-of-Expert (MoE) architectures with specialized experts and routing, they still inflate parameter counts and introduce routing overhead. In this paper, we propose Omni-C (Omni-Compress), a single dense Transformer-based encoder that learns competitive shared representations across heterogeneous modalities--images, audio, and text--through unimodal contrastive pretraining on large-scale unaligned data. By maximizing parameter sharing in the backbone and using lightweight modality-specific projection heads, Omni-C effectively mitigates inter-modality conflicts without requiring MoE, paired supervision, or routing. This design supports efficient deployment on memory-constrained systems via sequential modality processing and low-memory inference, eliminating the need for parallel expert loading or specialized hardware. Experiments show Omni-C achieves performance comparable to expert models in unimodal and cross-model tasks, with modest zero-shot degradation on audio and text that is largely recovered through lightweight linear probing or parameter efficient fine-tuning. The unified architecture substantially reduces inference memory usage compared to multi-encoder baselines, advancing efficient and scalable multimodal learning.
Kin Wai Lau, Yasar Abbas Ur Rehman, Lai-Man Po +1
City University of Hong Kong · TCL AI Lab · University of Surrey, United Kingdom
Recent advances in omni-modal large language models have enabled remarkable progress in joint vision-audio understanding. However, prevailing architectures rely on modality-specific encoders with a video-coarse, audio-dense design -- sampling visual frames at 1--2 fps while processing audio waveforms at 25 fps -- resulting in systems that perceive video frame by frame, modality by modality rather than holistically as humans do. Such a discrepancy leaves models with impoverished cross-modal interaction during encoding and an inability to capture fine-grained visual motion. To bridge this gap, we present Omni-Encoder, a unified Transformer backbone designed to co-embed visual and audio signals at a symmetrical 25 fps within a shared latent space. This architecture leverages three core innovations -- the Omni-Encoder Token Template, Omni-RoPE, and Temporal Window Shifting -- to effectively reconcile the dual challenges of modality disentanglement and computational efficiency. Experiments demonstrate that, compared to the modality-specific baseline Qwen2.5-Omni under the same input token budget to the LLM decoder, Omni-Encoder delivers substantial gains on visual continuous understanding tasks -- such as sign language recognition and fine-grained sports action analysis -- while maintaining competitive performance on established audio-visual benchmarks such as AVQA and Speaker Identification and Localization. These results suggest that unified omnivorous encoding offers a promising direction for building omni-modal models that more closely reflect the integrated nature of human perception.
Detao Bai, Shimin Yao, Weixuan Chen +4
1Tongyi Lab Alibaba Group · Shenzhen University of Advanced Technology
Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces
Vladan Stojnić, Ryan Ramos, Giorgos Kordopatis-Zilos +2
VRG, FEE, Czech Technical University in Prague · The University of Osaka