Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning

Abhishek Dalvi; Vasant Honavar

cs.CVFeb 27, 2026

Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning

Authors: Abhishek Dalvi, Vasant Honavar

Organizations: Artificial Intelligence Research Laboratory, The Pennsylvania State University

Abstract

Large unimodal foundation models for vision and language encode rich semantic structures, yet aligning them typically requires computationally intensive multimodal fine-tuning. Such approaches depend on large-scale parameter updates, are resource intensive, and can perturb pretrained representations. Emerging evidence suggests, however, that independently trained foundation models may already exhibit latent semantic compatibility, reflecting shared structures in the data they model. This raises a fundamental question: can cross-modal alignment be achieved without modifying the models themselves? Here we introduce HDFLIM (HyperDimensional computing with Frozen Language and Image Models), a framework that establishes cross-modal mappings while keeping pretrained vision and language models fully frozen. HDFLIM projects unimodal embeddings into a shared hyperdimensional space and leverages lightweight symbolic operations -- binding, bundling, and similarity-based retrieval to construct associative cross-modal representations in a single pass over the data. Caption generation emerges from high-dimensional memory retrieval rather than iterative gradient-based optimization. We show that HDFLIM achieves performance comparable to end-to-end vision-language training methods and produces captions that are more semantically grounded than zero-shot baselines. By decoupling alignment from parameter tuning, our results suggest that semantic mapping across foundation models can be realized through symbolic operations on hyperdimensional encodings of the respective embeddings. More broadly, this work points toward an alternative paradigm for foundation model alignment in which frozen models are integrated through structured representational mappings rather than through large-scale retraining. The codebase for our implementation can be found at https://github.com/Abhishek-Dalvi410/HDFLIM.

Figures & tables

Figure 1: Overview of the HDFLIM algorithm where Image Encoder and LLM are kept frozen. HD representations of the $i^{th}$ token are binded with the HD representation of the image, and this acts as an contextual cue for the next token during the inference stage.

Model	B@4	M	C	S	CLIP-S	RefCLIP-S
ZeroCap [ 60 ]	2.6	11.5	14.6	5.5	87.0	79.0
ConZIC [ 71 ]	1.3	11.2	13.2	5.0	99.0	83.0
MAGIC [ 58 ]	12.9	17.2	48.3	10.9	74.0	77.0
ViECap ⊤ [ 19 ]	26.7	24.0	89.6	17.5	75.6	-
CLIP-Captioner-B/32 [ 7 ]	36.0	27.8	114.9	20.8	75.0	80.3
CLIP-Captioner-L/14 [ 7 ]	38.7	29.3	126.0	22.5	75.4	81.1

Table 1: Performance on Karpathy Test Split of COCO. Max number of generated tokens for HDFLIM and Qwen2VL are limited to 18. ⊤ values were taken from Bianchi et al. [8]

	NoCaps Domains
Model	In		Near		Out		Overall		Overall
	C	S	C	S	C	S	C	S	CLIP-S
ZeroCap* [ 60 ]	13.3	-	14.9	-	19.7	-	16.6	-	-
ConZIC* [ 71 ]	13.7	-	15.8	-	18.3	-	16.9	-	-
MAGIC † [ 58 ]	-	-	-	-	-	-	-	-	66.2
CapDec † [ 45 ]	60.1	10.2	50.2	9.3	28.7	6.0	45.9	8.3	69.2

Table 2: Zero-shot performance on NoCaps Val split with CIDEr (C), SPICE (S) and CLIP-Scores metrics. In, Near and Out are Domains w.r.t to the objects/concepts with the MS-COCO dataset. * results were taken from Zeng et al. [70] and only contained CIDEr score. † values were taken from Fei et al. [19]

	HDFLIM (C)	HDFLIM (P)
BLEU@4	+17.6	+16.1
METEOR	+6.4	+8.1
CIDEr	+34.2	+43.5
SPICE	+3.8	+6.5
CLIP-S	-6.3	-3.6
RefCLIP-S	-3.6	-1.6

Table 3: Effect of BART post-processing on HDFLIM predictions for COCO. Values shown in green and red indicate improvements and degradations, respectively, for each metric.

Figure 2: Captioning performance of HDFLIM variants evaluated against the ZeroCap baseline (dotted line). HDFLIM show transferability from Base to Instruct Qwen4B variants, with modest performance degradation, when both variants are compared to ZeroCap.

Figure 3: Reference free metric evaluation on COCO. Max number of generated tokens for HDFLIM (P) are 41 with no prompt for paraphrasing. For Qwen2VL ${}_{\text{Base}}$ ; max generated tokens are 50 with prompt: "Provide a detailed description of this image, including the main subjects, their actions, the setting, and any notable details."

Figure 4: Inference Speed vs. Caption Length. These experiments were conducted on an NVIDIA A100 (40GB) GPU. ZeroCap and ConZIC were re-implemented for this study using the original hyperparameters reported in their respective publications to ensure fair benchmarking.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: HDFLIM (P) generated Long Caption: "Boy sleeping on a bed in his room, which is empty except for the mosquito net and some furniture. The background has an old wall with peeling paint that looks like a faded blue color."

Figure 6: HDFLIM (P) generated Long Caption: "Wet pavement in the rain. The people are wearing umbrellas and their faces can be seen, but they seem to have a lot of water on them or around themselves due the rain."

Figure 7: HDFLIM (C) generated Caption: "Birds fly over the water in a cloudy day." ; transforming this prediction using BART: "A flock of birds flying over the water."

Figure 8: HDFLIM(C) generated Caption: "Airport runway waiting planes in a terminal." ; transforming this prediction using BART: "A plane sitting on the tarmac at an airport."

Model	B@4	M	C	S	CLIP-S	RefCLIP-S
ZeroCap [ 60 ]	2.6	11.5	14.6	5.5	87.0	79.0
ConZIC [ 71 ]	1.3	11.2	13.2	5.0	99.0	83.0
HDFLIM (C) (w=1)	6.9	16.6	49.2	12.3	75.3	79.2
HDFLIM (C) (w=1) +BART	24.5	23.0	83.4	16.1	69.0	75.6
HDFLIM (C) (w=3)	5.4	15.2	43.3	11.3	74.8	79.0
HDFLIM (C) (w=3) +BART	24.3	22.9	82.9	16.2	69.0	75.4

Appendix

Table 4: Detailed Performance on Karpathy Test Split of COCO. Max number of generated tokens for HDFLIM and Qwen2VL are limited to 18. ⊤ values were taken from Bianchi et al. [8]

Model	B@4	M	C	S	CLIP-S	RefCLIP-S
No Training; Test time gradient compute or Sampling
ZeroCap [ 60 ]	2.6	11.5	14.6	5.5	87.0	79.0
ConZIC [ 71 ]	1.3	11.2	13.2	5.0	99.0	83.0
HDFLIM with Learning phase on COCO dataset
HDFLIM ${}_{\text{Qwen3-Base}}$	5.4	15.2	43.3	11.3	74.8	79.0
HDFLIM ${}_{\text{Qwen3-Instruct}}$	5.3	15.4	41.1	10.9	73.5	77.5

Appendix

Table 5: Performance of transferability of learned symbolic prototype across various Finetuned Version for Karpathy Test Split of COCO when Qwen3-4B Base model is replaced with Qwen3-4B-Instruct for Inference. Across all results, HDFLIM Learning was done on only Qwen3-4B base model. ZeroCap and ConZIC are kept as baselines. (w=3)

Model	CLIP-S	RefCLIP-S
HDFLIM (P) (w=3)	76.6	76.4
Qwen2VL ${}_{\text{Base}}$ [ 66 ]	77.5	77.9

Appendix

Table 6: Performance on Karpathy Test Split of COCO. Max number of generated tokens for HDFLIM are 41 and Qwen2VL is 50 with prompt: "Provide a detailed description of this image, including the main subjects, their actions, the setting, and any notable details."

Model	B@4	M	C	S	CLIP-S	RefCLIP-S
HDFLIM (C) (w=3)	6.3	12.8	26.5	8.7	73.1	75.8
MAGIC [ 58 ]	6.2	12.2	17.5	5.9	68.6	-
CapDec [ 45 ]	17.3	18.6	35.7	-	73.7	-
ViECap [ 19 ]	17.4	18.0	38.4	11.2	76.1	-
Qwen2VL ${}_{\text{FT}}$ [ 66 ]	32.1	25.5	88.3	18.5	78.7	82.3

Appendix

Table 7: Cross-Domain Performance on Flickr30K Karpathy Test split; Trained/Learned on COCO; and Tested on Flickr30K Karpathy Test split

CLIP-Weight	B@4	M	C	S	CLIP-S	RefCLIP-S
0.00	7.0	13.6	26.8	8.4	59.2	65.7
0.25	7.7	15.2	42.5	11.0	70.0	75.4
0.50	5.4	15.2	43.3	11.3	74.8	79.0
0.75	2.1	13.8	35.3	9.5	77.1	80.0
1.00	0.2	12.7	25.0	8.0	77.5	78.4

Appendix

Table 8: Performance of HDFLIM (C) with w=3 with varying Clip-weight parameter during sampling for Test Split of COCO dataset

W	B@4	M	C	S	CLIP-S	RefCLIP-S
1	6.9	16.6	49.2	12.3	75.3	79.2
2	6.2	15.8	46.1	11.7	75.0	79.0
3	5.4	15.2	43.3	11.3	74.8	79.0

Appendix

Table 9: Performance of HDFLIM (C) with varying window parameter during sampling for Karpathy Test Split of COCO dataset

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 440 papers

Mar 31, 2026

The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

Vision-Language Models (VLMs) such as CLIP learn a shared embedding space for images and text, yet their representations remain geometrically separated, a phenomenon known as the modality gap. This gap limits tasks requiring cross-modal interchangeability, such as captioning and joint clustering. Existing post-processing approaches can partially improve cross-modal compatibility; however, we show through geometric analysis that they primarily reduce the global centroid offset while leaving the underlying distributional mismatch intact. We decompose the modality gap into a Centroid Gap and a Distribution Gap, and demonstrate that the Distribution Gap is the true predictor of cross-modal task quality (

R^2 = 0.986

), whereas the commonly used Raw Gap is misleading (

R^2 = 0.691

). Motivated by this observation, we propose TPC-CMA (Three-Phase Curriculum for Cross-Modal Alignment), a fine-tuning framework that explicitly reduces both components. The proposed CMA jointly mitigates centroid offsets and reshapes the distributional structure, while a three-phase curriculum with gradient-aware scheduling progressively introduces alignment during training to enable stable optimization. Experiments demonstrate that our method significantly improves cross-modal alignment. With

α_{\text{target}}{=}0.05

, the modality gap is reduced by 66.6% with only 4.84% accuracy drop. Under stronger alignment (

α_{\text{target}}{=}0.5

), the gap is reduced by 82.3%, clustering ARI improves from 0.318 to 0.516, and captioning CIDEr increases by 57.1% over the original model. Our code and pre-trained models will be made publicly available upon acceptance.

Hongyuan Liu, Qinli Yang, Wen Li +6

University of Electronic Science and Technology of China · University of Bristol

Cross-Modal Alignment Modality Gap in VLMs VLM Adaptation

Ask PDF

Apr 6, 2026

The Indra Representation Hypothesis for Multimodal Alignment

Recent studies have uncovered an interesting phenomenon: unimodal foundation models tend to learn convergent representations, regardless of differences in architecture, training objectives, or data modalities. However, these representations are essentially internal abstractions of samples that characterize samples independently, leading to limited expressiveness. In this paper, we propose The Indra Representation Hypothesis, inspired by the philosophical metaphor of Indra's Net. We argue that representations from unimodal foundation models are converging to implicitly reflect a shared relational structure underlying reality, akin to the relational ontology of Indra's Net. We formalize this hypothesis using the V-enriched Yoneda embedding from category theory, defining the Indra representation as a relational profile of each sample with respect to others. This formulation is shown to be unique, complete, and structure-preserving under a given cost function. We instantiate the Indra representation using angular distance and evaluate it in cross-model and cross-modal scenarios involving vision, language, and audio. Extensive experiments demonstrate that Indra representations consistently enhance robustness and alignment across architectures and modalities, providing a theoretically grounded and practical framework for training-free alignment of unimodal foundation models. Our code is available at https://github.com/Jianglin954/Indra.

Jianglin Lu, Hailing Wang, Kuo Yang +3

Department of Electrical and Computer Engineering, Northeastern University · Adobe Research · Khoury College of Computer Science, Northeastern University

Representation Learning Cross-Modal Alignment Cross-Modal Representation Learning

Ask PDF

Mar 3, 2026

Beyond Language Modeling: An Exploration of Multimodal Pretraining

The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.

Shengbang Tong, David Fan, John Nguyen +18

FAIR, Meta · New York University

Multimodal Pretraining Unified Multimodal Models Scaling Laws

Ask PDF

Abstract

Cross-Modal Alignment

Explore similar work

The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

The Indra Representation Hypothesis for Multimodal Alignment

Beyond Language Modeling: An Exploration of Multimodal Pretraining