cs.LGNov 16, 2026

Learning What Pairs Miss for Higher-Order Multimodal Alignment

Authors: Kunyu Zhang, Shenghui Feng, Xiangjie Shi, Zeqing Wang, Yingxu Wang

Organizations: EBioMed · JLU · USTB · PolyU HK · 

Abstract

Contrastive learning beyond two modalities now scores the whole tuple with multilinear products, volumes, and fusion heads, critics that can represent the joint distribution. In practice these critics still behave like bags of pairs. Their negatives break at least two of the three pairs, a single pairwise check rejects them, and when most samples can be solved by pairs the higher-order term receives almost no gradient. To solve this problem, we propose CHORD (Contrastive Higher-Order Residuals with Decoys), which trains a residual against decoys drawn from a calibrated pairwise model, negatives that the pairwise model finds plausible, so that rejecting them often needs all three modalities. Drawing decoys costs one inner-product query over a memory bank, and the optimal residual is the log-ratio between the data and the pairwise model, so the decoy loss also bounds what that model misses. On M3-Mix, where one in ten training samples needs all three modalities, our CHORD improves accuracy on those samples by 21.2 percentage points over the strongest of eight baselines. CHORD also gives the best two-to-one results on every real dataset, and its residual improves other retrievers without retraining.

Figures & tables

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment

    May 28, 2026Tianchao Li, Shujian Yu, Xinrui Zu +4Cross-Modal LearningCross-Modal Alignment

  2. SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

    Sep 26, 2026Yavuz Yarici, Ghassan AlRegibCross-Modal Representation LearningMultimodal Contrastive Learning

  3. Multimodal LLMs under Pairwise Modalities

    May 20, 2026Yan Li, Yunlong Deng, Yuewen Sun +3Cross-Modal LearningRepresentation Identifiability