Cross-Modal Representation Learning
Overview
The field studies how models can represent different kinds of data so that related observations can be matched, combined, or used to transfer knowledge. Common approaches include CLIP-style contrastive alignment, masked cross-modal reconstruction and predictive learning, teacher–student distillation, and methods that separate shared from modality-specific features; anchors and optimal transport can help when paired examples are scarce. These representations support image-text retrieval, cross-sensor recognition, and transfer across domains or missing-modality settings. A central tension is that shared spaces can improve transfer but discard useful modality-specific structure, and apparent semantic alignment does not always imply fine-grained correspondence; the best choice between alignment and prediction depends on data quality and cross-modal relationships.
Latest papers
224 papers17 papers in the last four weeks, down 15% on the four weeks before. 0.1% of all new papers.
CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension
MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignment
SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery
Semantic Purification for Conditional Representation Learning
Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation
perception'' models that understand motion from video but only output text, and generation'' models that cannot perceive from raw visual input. Second, generative MLLMs are often limited to single-frame, static poses using dense, parametric SMPL models, failing to handle temporal motion. Third, existing motion vocabularies are built from skeleton data alone, severing the link to the visual domain. To address these challenges, we introduce Superman, a unified framework that bridges visual perception with temporal, skeleton-based motion generation. Our solution is twofold. First, to overcome the modality disconnect, we propose a Vision-Guided Motion Tokenizer. Leveraging the natural geometric alignment between 3D skeletons and visual data, this module pioneers robust joint learning from both modalities, creating a unified, cross-modal motion vocabulary. Second, grounded in this motion language, a single, unified MLLM architecture is trained to handle all tasks. This module flexibly processes diverse, temporal inputs, unifying 3D skeleton pose estimation from video (perception) with skeleton-based motion prediction and in-betweening (generation). Extensive experiments on standard benchmarks, including Human3.6M, demonstrate that our unified method achieves state-of-the-art or competitive performance across all motion tasks. This showcases a more efficient and scalable path for generative motion analysis using skeletons.