Multimodal learning commonly assumes paired or synchronized observations, yet independently collected datasets rarely provide such correspondence. We introduce CIIR, a modular encoder--decoder framework that connects heterogeneous datasets through a class-indexed address space while preserving each dataset's native input format and output module. A known local index selects address k, and an input-dependent 128-dimensional address representation Z_k(x) is written to that address. To demonstrate CIIR, we learn this address space from handwritten digits, clothing images, spoken-digit spectrograms, and hand signs without synchronized or instance-matched samples. Newly trained label-free receivers recover the requested addresses with 94.7% macro-average Address Realization Accuracy. We then attach the unseen SVHN dataset after freezing the base address space and train only lightweight adapters, achieving 90.2% MNIST>SVHN and 82.3% SVHN>MNIST end-to-end Address Transfer Accuracy without target-side decoder labels. To demonstrate generalizability of CIIR, we reuse the ten addresses as code elements in collision-free arithmetic and hierarchical address codes for all 100 CIFAR-100 classes and evaluate direct arithmetic-code realization through the existing receivers. These results establish CIIR as a compact interface for modular cross-dataset communication and post-hoc dataset attachment without exhaustive paired multimodal supervision.
Figures & tables
Figure 1: Qualitative realization of the ten CIIR addresses via their original label-conditioned decoders. Each pair of rows shows ground truth and generated outputs. Equal local indices select shared addresses while preserving dataset-local classes. For independent label-free evidence, see Table 1 .
Figure 2: Compositional application of CIIR to class-count mismatch across independently collected modalities. The ten learned addresses are reused as a coding vocabulary for the 100 CIFAR-100 classes: a known fine class is mapped to (a) the arithmetic address code [7,2] or (b) the hierarchical address code [1,0,3] , and each requested address is realized in the native output domain of each modality.
Dataset
Representation
ARA
SSIM
L1
Div. Ret.
MNIST
Input-dependent Zk(x)
0.9855
0.8434
0.0435
0.9055
MNIST prototype pk
1.0000
0.6511
0.1097
0.0000
Fashion-MNIST
Input-dependent Zk(x)
0.9343
0.6943
0.0597
0.8609
MNIST prototype pk
0.8000
0.3498
0.2238
0.0000
FSDD
Input-dependent Zk(x)
0.8911
0.7518
0.0420
0.8552
MNIST prototype pk
0.4000
0.2066
0.2192
0.0000
Table 1: Label-free receiver evaluation. Input-dependent address representations are compared with a fixed MNIST-derived address-prototype bank. Address Realization Accuracy (ARA) measures realization of the requested address. Div. Ret. denotes diversity retention (generated within-class variance relative to real test data; 0.0000 indicates that a fixed prototype generates essentially the same output for every sample assigned to that address).
Dataset
Representation
ARA
SSIM
L1
Div. Ret.
MNIST
Input-dependent Zk(x)
0.7730
0.6566
0.1051
0.1550
Modality prototype pk(m)
0.9018
0.6474
0.1097
0.0000
Fashion-MNIST
Input-dependent Zk(x)
0.9752
0.5283
0.1067
0.3412
Modality prototype pk(m)
1.0000
0.4755
0.1451
0.0000
FSDD
Input-dependent Zk(x)
0.8444
0.4265
0.1036
0.2117
Modality prototype pk(m)
0.8000
0.3549
0.1330
0.0000
Table 2: Original label-conditioned decoder evaluation. Input-dependent address representations are compared with modality-specific address prototypes. Because the known address k is supplied in both conditions, SSIM, L1, and diversity retention provide the primary evidence of observation-specific information. Div. Ret. denotes diversity retention relative to real test data; 0.0000 indicates that a fixed prototype produces essentially the same output for every sample assigned to that address.
Rep.
ARA
SSIM
L1
Div. Ret.
CIIR 1
0.9855
0.8434
0.0435
0.9055
FV-FR 2
0.7698±0.1104
0.6771±0.0013
0.1052±0.0012
≈0
FV-TR 3
0.8397±0.0616
0.6759±0.0044
0.1069±0.0028
≈0
Table 3: Learned fixed address-vector baselines on MNIST. CIIR uses an input-dependent address representation, whereas FV-FR and FV-TR use one fixed learned vector per address. CIIR reports the saved-model result; the baselines report mean ± standard deviation over three seeds.
Model
Intermediate Representation
Direction
Metric
Score
MVAE 1
PoE latent
MNIST → SVHN
Digit matching
9.5%
PoE latent
SVHN → MNIST
Digit matching
9.3%
MMVAE 2
MoE shared latent
MNIST → SVHN
Digit matching
86.4%
MoE shared latent
SVHN → MNIST
Digit matching
69.1%
mmJSD 3
JSD-aligned latent
MST → missing modalities
Conditional coherence
0.72
DMVAE 4
Shared/private latent
MNIST → SVHN
Cross-synthesis accuracy
88.1%
Table 4: Reported results on the MNIST–SVHN–Text multimodal representation-learning benchmark ( Sutter et al. 2020 ) . Prior methods report their original evaluation metrics, while CIIR reports End-to-end Address Transfer Accuracy and target-side ARA through the source → address → target path.
Figure 3: Real SVHN examples and label-free outputs generated from the ten frozen CIIR address prototypes. The SVHN decoder receives no target-side class label.
Code
Receiver
RT
Collision
ARA
CRA
Arithmetic
MNIST
1.000
0
1.000
1.000
F-MNIST
1.000
0
0.800
0.640
FSDD
1.000
0
0.400
0.160
Sign-MNIST
1.000
0
0.100
0.010
Hierarchical
Structure
1.000
0
–
–
Table 5: CIFAR-100 address-code evaluation. RT and Collision describe the deterministic code structure. ARA measures realization of individual addresses, and CRA requires every address in a class code to be realized correctly.
Table 6: Architectures of the principal CIIR components. All convolutional kernels are 3×3 unless otherwise specified. GAP denotes global average pooling, BN denotes batch normalization, and LReLU uses slope 0.2.
Stage
Optimizer
Learning rate
Batch
Maximum epochs
Early stopping
MNIST private AE
Adam
2×10−4
64
100
min 40, patience 20
Fashion private AE
Adam
1×10−4
64
120
min 40, patience 20
FSDD private AE
Adam
1×10−4
64
180
min 60, patience 30
Sign private AE
Adam
2×10−4
64
100
min 40, patience 20
Unified encoder
Adam
1×10−4
64
200
min 80, patience 25
Exploratory CIFAR encoder
Adam
2×10−4
16
80
min 30, patience 15
Table 7: Training and model hyperparameters used in the reported runs.
Dataset
Original
Perm. → Target
Perm. → Original
Zero
Random
MNIST
0.986
0.985
0.001
0.098
0.099
Fashion-MNIST
0.934
0.933
0.002
0.100
0.099
FSDD
0.907
0.896
0.004
0.100
0.109
Sign Language MNIST
0.976
0.975
0.000
0.100
0.106
Macro average
0.951
0.947
0.002
0.100
0.103
Table 8: Address-representation substitution results. All columns report ARA; chance is 0.10 .
Direction
Address Routing
Target ARA
End-to-end Transfer
Transfer Retention
MNIST → SVHN
0.9095
0.9918
0.9024
0.9922
SVHN → MNIST
0.8229
1.0000
0.8229
1.0000
Table 9: Internal CIIR decomposition for bidirectional MNIST–SVHN transfer.
Table 11: Direct arithmetic address-code realization using the frozen MNIST-derived address prototypes. k1 ARA and k2 ARA report realization at each code position; ARA averages both positions, and CRA requires both to be correct.
Receiver
k1 ARA
k2 ARA
ARA
CRA
MNIST
0.2699
0.1599
0.2149
0.0400
Fashion-MNIST
0.5086
0.4000
0.4543
0.2000
FSDD
0.0000
0.2035
0.1018
0.0000
Sign Language MNIST
0.3000
0.2371
0.2686
0.0900
Table 12: Exploratory CIFAR-conditioned generation of arithmetic-code address representations. k1 ARA and k2 ARA report each code position; ARA averages both positions, and CRA requires both to be correct.
Figure 4: Extended qualitative comparison of the ten CIIR addresses, grouped by output modality. For each modality and local index, two real examples and two generated examples are shown. Generated outputs use the two actual MNIST address representations nearest to the corresponding address centroid, together with the requested local index, through the original frozen dataset-specific decoder. The use of address-centroid-nearest representations provides a deterministic representative selection procedure rather than manual visual selection.
Figure 5: Extended arithmetic two-address code realizations grouped by modality. Each example maps a known CIFAR-100 fine-class identity c to the ordered two-address decimal code ϕarith(c)=[⌊c/10⌋,cmod10] . For every modality and target class, the first two columns show real examples associated with the two addresses, while the final two columns show their generated realizations through the corresponding frozen label-conditioned decoder. The figure visualizes reuse of the ten addresses as an ordered class-code vocabulary rather than CIFAR-100 image generation. This qualitative decoder visualization is distinct from the label-free receiver evaluation reported in Table 11 .
Figure 6: Extended hierarchical three-address code realizations grouped by modality. For a CIFAR-100 fine class c , the code ϕhier(c)=[⌊s(c)/10⌋,s(c)mod10,m(c)] represents its superclass s(c) using the first two addresses and its within-superclass member address m(c) using the third. For each modality, the first three columns show real examples associated with the requested addresses and the final three columns show their generated realizations. Examples belonging to the same superclass preserve a common two-address prefix, illustrating hierarchical organization through ordered address reuse. This qualitative decoder visualization illustrates the deterministic code structure; quantitative hierarchical execution was not evaluated.
In-Context Learning (ICL) is a significant paradigm for Large Multimodal Models (LMMs), using a few in-context demonstrations (ICDs) for new task adaptation. However, its performance is sensitive to demonstration configurations and computationally expensive. Mathematically, the influence of these demonstrations can be decomposed into a dynamic mixture of the standard attention output and the context values. Current approximation methods simplify this process by learning a "shift vector". Inspired by the exact decomposition, we introduce High-Fidelity In-Context Learning (HIFICL) to more faithfully model the ICL mechanism. HIFICL consists of three key components: 1) a set of "virtual key-value pairs" to act as a learnable context, 2) a low-rank factorization for stable and regularized training, and 3) a simple end-to-end training objective. From another perspective, this mechanism constitutes a form of context-aware Parameter-Efficient Fine-Tuning (PEFT). Extensive experiments show that HiFICL consistently outperforms existing approximation methods on several multimodal benchmarks. The code is available at https://github.com/bbbandari/HiFICL.
Xiaoyu Li, Yuhang Liu, Xuanshuo Kang +4
University of Electronic Science and Technology of China · Institute of Electronics and Information Industry Technology of Kashgar
Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks. Despite its flexibility, multimodal ICL incurs high inference latency and suffers from instability due to sensitivity to demonstration formatting, ordering, and content. To address these limitations, we propose Hyper-ICL, a lightweight, training-based framework for demonstration-free multimodal ICL that reconstructs demonstration effects directly without requiring ICDs at inference time. Hyper-ICL learns a parameter-efficient low-rank logit-level adapter that calibrates attention distributions to better match demonstration-induced attention redistribution. To capture how demonstration influence varies across queries, we introduce a query-adaptive modulation mechanism that adaptively controls intervention strength at token level across layers and heads based on the current query. Finally, we propose a layer-wise hyperbolic anchor distillation loss that aligns intermediate student features to a demonstration-conditioned teacher via Lorentz geodesic distance. This loss encourages the student to reconstruct the demonstration-query relationships induced by ICDs. Extensive experiments across six different multimodal benchmarks (including VQAv2, OK-VQA, and COCO Caption) demonstrate that Hyper-ICL consistently improves accuracy and stability over vanilla ICL and existing state-of-the-art methods.
Composed Image Retrieval (CIR) has attracted significant attention due to its flexible multimodal query method, yet its development is severely constrained by the Noisy Triplet Correspondence (NTC) problem. Most existing robust learning methods rely on the "small loss hypothesis", but the unique semantic ambiguity in NTC, such as "partial matching", invalidates this assumption, leading to unreliable noise identification. This entraps the model in a self dependent vicious cycle where the learner is intertwined with the arbiter, ultimately causing catastrophic "representation pollution". To address this critical challenge, we propose a novel "Expert-Proxy-Diversion" decoupling paradigm, named Air-Know (ArbIteR calibrated Knowledge iNternalizing rObust netWork). Air-Know incorporates three core modules: (1) External Prior Arbitration (EPA), which utilizes Multimodal Large Language Models (MLLMs) as an offline expert to construct a high precision anchor dataset; (2) Expert Knowledge Internalization (EKI), which efficiently guides a lightweight proxy "arbiter" to internalize the expert's discriminative logic; (3) Dual Stream Reconciliation (DSR), which leverages the EKI's matching confidence to divert the training data, achieving a clean alignment stream and a representation feedback reconciliation stream. Extensive experiments on multiple CIR benchmark datasets demonstrate that Air-Know significantly outperforms existing SOTA methods under the NTC setting, while also showing strong competitiveness in traditional CIR.