CIIR - Class-Indexed Interface Representation Learning

Doo Yon Kim; Christian Wallraven

cs.AIOct 16, 2026

CIIR - Class-Indexed Interface Representation Learning

Authors: Doo Yon Kim, Christian Wallraven

Abstract

Multimodal learning commonly assumes paired or synchronized observations, yet independently collected datasets rarely provide such correspondence. We introduce CIIR, a modular encoder--decoder framework that connects heterogeneous datasets through a class-indexed address space while preserving each dataset's native input format and output module. A known local index selects address $k$ , and an input-dependent 128-dimensional address representation Z_k(x) is written to that address. To demonstrate CIIR, we learn this address space from handwritten digits, clothing images, spoken-digit spectrograms, and hand signs without synchronized or instance-matched samples. Newly trained label-free receivers recover the requested addresses with 94.7% macro-average Address Realization Accuracy. We then attach the unseen SVHN dataset after freezing the base address space and train only lightweight adapters, achieving 90.2% MNIST>SVHN and 82.3% SVHN>MNIST end-to-end Address Transfer Accuracy without target-side decoder labels. To demonstrate generalizability of CIIR, we reuse the ten addresses as code elements in collision-free arithmetic and hierarchical address codes for all 100 CIFAR-100 classes and evaluate direct arithmetic-code realization through the existing receivers. These results establish CIIR as a compact interface for modular cross-dataset communication and post-hoc dataset attachment without exhaustive paired multimodal supervision.

Figures & tables

Figure 1: Qualitative realization of the ten CIIR addresses via their original label-conditioned decoders. Each pair of rows shows ground truth and generated outputs. Equal local indices select shared addresses while preserving dataset-local classes. For independent label-free evidence, see Table 1 .

Panel of Figure 2 — Figure 2: Compositional application of CIIR to class-count mismatch across independently collected modalities. The ten learned addresses are reused as a coding vocabulary for the 100 CIFAR-100 classes: a known fine class is mapped to (a) the arithmetic address code $[7,2]$ or (b) the hierarchical address code $[1,0,3]$ , and each requested address is realized in the native output domain of each modality.

Dataset	Representation	ARA	SSIM	L1	Div. Ret.
MNIST	Input-dependent $Z_{k}(x)$	0.9855	0.8434	0.0435	0.9055
	MNIST prototype $p_{k}$	1.0000	0.6511	0.1097	0.0000
Fashion-MNIST	Input-dependent $Z_{k}(x)$	0.9343	0.6943	0.0597	0.8609
	MNIST prototype $p_{k}$	0.8000	0.3498	0.2238	0.0000
FSDD	Input-dependent $Z_{k}(x)$	0.8911	0.7518	0.0420	0.8552
	MNIST prototype $p_{k}$	0.4000	0.2066	0.2192	0.0000

Table 1: Label-free receiver evaluation. Input-dependent address representations are compared with a fixed MNIST-derived address-prototype bank. Address Realization Accuracy (ARA) measures realization of the requested address. Div. Ret. denotes diversity retention (generated within-class variance relative to real test data; 0.0000 indicates that a fixed prototype generates essentially the same output for every sample assigned to that address).

Dataset	Representation	ARA	SSIM	L1	Div. Ret.
MNIST	Input-dependent $Z_{k}(x)$	0.7730	0.6566	0.1051	0.1550
	Modality prototype $p_{k}^{(m)}$	0.9018	0.6474	0.1097	0.0000
Fashion-MNIST	Input-dependent $Z_{k}(x)$	0.9752	0.5283	0.1067	0.3412
	Modality prototype $p_{k}^{(m)}$	1.0000	0.4755	0.1451	0.0000
FSDD	Input-dependent $Z_{k}(x)$	0.8444	0.4265	0.1036	0.2117
	Modality prototype $p_{k}^{(m)}$	0.8000	0.3549	0.1330	0.0000

Table 2: Original label-conditioned decoder evaluation. Input-dependent address representations are compared with modality-specific address prototypes. Because the known address

k

is supplied in both conditions, SSIM, L1, and diversity retention provide the primary evidence of observation-specific information. Div. Ret. denotes diversity retention relative to real test data; 0.0000 indicates that a fixed prototype produces essentially the same output for every sample assigned to that address.

Rep.	ARA	SSIM	L1	Div. Ret.
CIIR 1	0.9855	0.8434	0.0435	0.9055
FV-FR 2	$0.7698{\pm}0.1104$	$0.6771{\pm}0.0013$	$0.1052{\pm}0.0012$	$\approx 0$
FV-TR 3	$0.8397{\pm}0.0616$	$0.6759{\pm}0.0044$	$0.1069{\pm}0.0028$	$\approx 0$

Table 3: Learned fixed address-vector baselines on MNIST. CIIR uses an input-dependent address representation, whereas FV-FR and FV-TR use one fixed learned vector per address. CIIR reports the saved-model result; the baselines report mean

\pm

standard deviation over three seeds.

Model	Intermediate Representation	Direction	Metric	Score
MVAE 1	PoE latent	MNIST $\rightarrow$ SVHN	Digit matching	9.5%
	PoE latent	SVHN $\rightarrow$ MNIST	Digit matching	9.3%
MMVAE 2	MoE shared latent	MNIST $\rightarrow$ SVHN	Digit matching	86.4%
	MoE shared latent	SVHN $\rightarrow$ MNIST	Digit matching	69.1%
mmJSD 3	JSD-aligned latent	MST $\rightarrow$ missing modalities	Conditional coherence	0.72
DMVAE 4	Shared/private latent	MNIST $\rightarrow$ SVHN	Cross-synthesis accuracy	88.1%

Table 4: Reported results on the MNIST–SVHN–Text multimodal representation-learning benchmark ( Sutter et al. 2020 ) . Prior methods report their original evaluation metrics, while CIIR reports End-to-end Address Transfer Accuracy and target-side ARA through the source

\rightarrow

address

\rightarrow

target path.

Figure 3: Real SVHN examples and label-free outputs generated from the ten frozen CIIR address prototypes. The SVHN decoder receives no target-side class label.

Code	Receiver	RT	Collision	ARA	CRA
Arithmetic	MNIST	1.000	0	1.000	1.000
	F-MNIST	1.000	0	0.800	0.640
	FSDD	1.000	0	0.400	0.160
	Sign-MNIST	1.000	0	0.100	0.010
Hierarchical	Structure	1.000	0	–	–

Table 5: CIFAR-100 address-code evaluation. RT and Collision describe the deterministic code structure. ARA measures realization of individual addresses, and CRA requires every address in a class code to be realized correctly.

Module	Architecture	Output
Private encoder	Conv-64/2–LReLU–BN; Conv-128/2–LReLU–BN; Conv-256/2–LReLU–BN; GAP; FC-256-ReLU	128-D
Original decoder	$[h^{(m)};\mathrm{Emb}_{64}(k)]$ ; FC- $(H/4)(W/4)256$ ; Deconv-256/2; Deconv-128/2; Conv-128–Conv-64–Conv-32	$H\times W\times C$
Unified encoder	Conv-64; Conv-64/2; Conv-128; Conv-128/2; Conv-256/2; GAP; FC-512; FC-256; 10 independent FC-256 address heads	$10\times 128$ representations + 10 logits
MNIST label-free receiver	FC- $7{\times}7{\times}256$ ; Deconv-256/2; Deconv-128/2; Conv-64	$28\times 28\times 1$
SVHN encoder	Conv-64/2; Conv-128/2; Conv-256/2; GAP; FC-512; FC-256; 10 independent FC-256 address heads	$10\times 128$ representations + 10 logits
SVHN label-free receiver	FC- $4{\times}4{\times}256$ ; Deconv-256/2; Deconv-128/2; Deconv-64/2; Conv-64	$32\times 32\times 3$

Table 6: Architectures of the principal CIIR components. All convolutional kernels are

3\times 3

unless otherwise specified. GAP denotes global average pooling, BN denotes batch normalization, and LReLU uses slope 0.2.

Stage	Optimizer	Learning rate	Batch	Maximum epochs	Early stopping
MNIST private AE	Adam	$2{\times}10^{-4}$	64	100	min 40, patience 20
Fashion private AE	Adam	$1{\times}10^{-4}$	64	120	min 40, patience 20
FSDD private AE	Adam	$1{\times}10^{-4}$	64	180	min 60, patience 30
Sign private AE	Adam	$2{\times}10^{-4}$	64	100	min 40, patience 20
Unified encoder	Adam	$1{\times}10^{-4}$	64	200	min 80, patience 25
Exploratory CIFAR encoder	Adam	$2{\times}10^{-4}$	16	80	min 30, patience 15

Table 7: Training and model hyperparameters used in the reported runs.

Dataset	Original	Perm. $\rightarrow$ Target	Perm. $\rightarrow$ Original	Zero	Random
MNIST	0.986	0.985	0.001	0.098	0.099
Fashion-MNIST	0.934	0.933	0.002	0.100	0.099
FSDD	0.907	0.896	0.004	0.100	0.109
Sign Language MNIST	0.976	0.975	0.000	0.100	0.106
Macro average	0.951	0.947	0.002	0.100	0.103

Table 8: Address-representation substitution results. All columns report ARA; chance is

0.10

Direction	Address Routing	Target ARA	End-to-end Transfer	Transfer Retention
MNIST $\rightarrow$ SVHN	0.9095	0.9918	0.9024	0.9922
SVHN $\rightarrow$ MNIST	0.8229	1.0000	0.8229	1.0000

Table 9: Internal CIIR decomposition for bidirectional MNIST–SVHN transfer.

Original	Perm. $\rightarrow$ Target	Perm. $\rightarrow$ Original	Zero	Random
0.9918	0.9902	0.0034	0.0958	0.0928

Table 10: MNIST

\rightarrow

SVHN address-space intervention. Chance is

0.10

Receiver	$k_{1}$ ARA	$k_{2}$ ARA	ARA	CRA
MNIST	1.0000	1.0000	1.0000	1.0000
Fashion-MNIST	0.8000	0.8000	0.8000	0.6400
FSDD	0.4000	0.4000	0.4000	0.1600
Sign Language MNIST	0.1000	0.1000	0.1000	0.0100

Table 11: Direct arithmetic address-code realization using the frozen MNIST-derived address prototypes.

k_{1}

ARA and

k_{2}

ARA report realization at each code position; ARA averages both positions, and CRA requires both to be correct.

Receiver	$k_{1}$ ARA	$k_{2}$ ARA	ARA	CRA
MNIST	0.2699	0.1599	0.2149	0.0400
Fashion-MNIST	0.5086	0.4000	0.4543	0.2000
FSDD	0.0000	0.2035	0.1018	0.0000
Sign Language MNIST	0.3000	0.2371	0.2686	0.0900

Table 12: Exploratory CIFAR-conditioned generation of arithmetic-code address representations.

k_{1}

ARA and

k_{2}

ARA report each code position; ARA averages both positions, and CRA requires both to be correct.

Figure 4: Extended qualitative comparison of the ten CIIR addresses, grouped by output modality. For each modality and local index, two real examples and two generated examples are shown. Generated outputs use the two actual MNIST address representations nearest to the corresponding address centroid, together with the requested local index, through the original frozen dataset-specific decoder. The use of address-centroid-nearest representations provides a deterministic representative selection procedure rather than manual visual selection.

Figure 5: Extended arithmetic two-address code realizations grouped by modality. Each example maps a known CIFAR-100 fine-class identity $c$ to the ordered two-address decimal code $\phi_{\mathrm{arith}}(c)=[\lfloor c/10\rfloor,c\bmod 10]$ . For every modality and target class, the first two columns show real examples associated with the two addresses, while the final two columns show their generated realizations through the corresponding frozen label-conditioned decoder. The figure visualizes reuse of the ten addresses as an ordered class-code vocabulary rather than CIFAR-100 image generation. This qualitative decoder visualization is distinct from the label-free receiver evaluation reported in Table 11 .

Figure 6: Extended hierarchical three-address code realizations grouped by modality. For a CIFAR-100 fine class $c$ , the code $\phi_{\mathrm{hier}}(c)=[\lfloor s(c)/10\rfloor,s(c)\bmod 10,m(c)]$ represents its superclass $s(c)$ using the first two addresses and its within-superclass member address $m(c)$ using the third. For each modality, the first three columns show real examples associated with the requested addresses and the final three columns show their generated realizations. Examples belonging to the same superclass preserve a common two-address prefix, illustrating hierarchical organization through ordered address reuse. This qualitative decoder visualization illustrates the deterministic code structure; quantitative hierarchical execution was not evaluated.

Cross-Modal Representation Learning

Cross-modal representation learning connects information across data types such as images, text, audio, biosignals, and sensor measurements.

See all 200 papers

HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks
Mar 13, 2026Xiaoyu Li, Yuhang Liu, Xuanshuo Kang +4Multimodal ICL Multimodal Large Language Models
University of Electronic Science and Technology of China · Institute of Electronics and Information Industry Technology of Kashgar
Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning
Jun 3, 2026Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh AfghahEfficient Multimodal Inference Multimodal ICL
Holcombe Department of Electrical and Computer Engineering, Clemson University, Clemson, United States.
Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrieval
Apr 21, 2026Zhiheng Fu, Yupeng Hu, Qianyun Yang +3Cross-Modal Alignment Learning with Noisy Labels
Shandong University

Mar 13, 2026J/K move · Enter open · S save

HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks

Xiaoyu Li, Yuhang Liu, Xuanshuo Kang, Zheng Luo+3

University of Electronic Science and Technology of China · Institute of Electronics and Information Industry Technology of Kashgar

Multimodal ICL Multimodal Large Language Models Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

In-Context Learning (ICL) is a significant paradigm for Large Multimodal Models (LMMs), using a few in-context demonstrations (ICDs) for new task adaptation. However, its performance is sensitive to demonstration configurations and computationally expensive. Mathematically, the influence of these demonstrations can be decomposed into a dynamic mixture of the standard attention output and the context values. Current approximation methods simplify this process by learning a "shift vector". Inspired by the exact decomposition, we introduce High-Fidelity In-Context Learning (HIFICL) to more faithfully model the ICL mechanism. HIFICL consists of three key components: 1) a set of "virtual key-value pairs" to act as a learnable context, 2) a low-rank factorization for stable and regularized training, and 3) a simple end-to-end training objective. From another perspective, this mechanism constitutes a form of context-aware Parameter-Efficient Fine-Tuning (PEFT). Extensive experiments show that HiFICL consistently outperforms existing approximation methods on several multimodal benchmarks. The code is available at https://github.com/bbbandari/HiFICL.

Abstract

Cross-Modal Representation Learning

Explore similar work

HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks

Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning

Air-Know: Arbiter-Calibrated Knowledge-Internalizing Robust Network for Composed Image Retrieval