cs.AINov 10, 2026

Integrating Action Prefix Prediction and Alignment Across Modalities for Multimodal Reinforcement Learning

Authors: Sijie Xu, Francisco Cruz

Organizations: School of Computer Science and Engineering, UNSW Sydney, Australia

Abstract

Learning representations from multimodal observations is difficult in reinforcement learning with limited interaction data. Multimodal Joint Predictive Representation combines multimodal latent prediction with contrastive learning across modalities, but its recursive rollout depends on earlier predictions and its pairwise alignment focuses on correspondence between modalities. We propose Improved Predictive Representation, which combines prediction from action prefixes with alignment at the observed anchor based on Contrastive Multimodal learning. At each horizon, it predicts future representations directly from the observed anchor and the corresponding sequence of actions. We evaluate two crop configurations: IPR-S with a minimum retained crop area of 10%, and IPR-H with 50%. Across five DeepMind Control Suite tasks, IPR-H achieves an average return across tasks that is 10.0% higher than the baseline with visual and depth images and 3.4% higher with visual images and position vectors. The relative performance of the two configurations varies across tasks. In Webots, both IPR configurations achieve higher mean final returns than the baseline without LiDAR, but not when LiDAR is available.

Figures & tables

Explore similar work

CardsList
  1. Towards Predictive, Aligned, and Scalable Robot Learning

    Jul 13, 2026Peijun Tang, Shangjin Xie, Baifu Huang +6Latent Action LearningCross-Modal Alignment

  2. Multimodal LLMs under Pairwise Modalities

    May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Cross-Modal LearningRepresentation Identifiability

  3. LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations

    May 1, 2026Huangbiao Xu, Huanqi Wu, Xiao Ke +1Missing-Modality LearningMultimodal Reasoning