AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models

Konstantinos D. Polyzos; Eleni Oikonomou; Tara Javidi

cs.LGSep 28, 2026

AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models

Authors: Konstantinos D. Polyzos, Eleni Oikonomou, Tara Javidi

Organizations: University of California San Diego

Abstract

Large foundation models have been introduced with the promise of efficient adaptation to downstream tasks. Yet, under limited supervision, MLLMs, an important class of large foundation models, remain challenging to adapt to various downstream tasks. Adaptation typically relies either on MLLM parameter fine-tuning or on training neural-based decoders. Both approaches struggle under limited supervision, while fine-tuning additionally requires access to model parameters, which is often unavailable for closed-source models. We introduce AdaKerNet, a novel learnable task-adaptive neural kernel decoder. AdaKerNet is fully agnostic to the parameters of the underlying MLLM and operates solely on its (frozen) rich representations obtained from the diverse available modalities. AdaKerNet relies on (i) a set of learnable, Lipschitz-controlled multimodal features derived from these MLLM representations; (ii) a reference kernel that provides a soft structural prior on those features; and (iii) a lightweight nonlinear neural predictor that adaptively deforms that structure. Learning the kernel representation and the neural predictor jointly within a unified optimization framework allows AdaKerNet to capture features and geometric relationships relevant to the downstream task. Numerical tests across four MLLMs: BLIP-2, LLaVA-1.5, Qwen2.5-VL, and Gemini Embedding 2, and multimodal inputs spanning text, audio, images, and tabular measurements demonstrate significant and consistent improvements over direct MLP, attention-, autoencoder- and kernel-based decoders, across a range of scarce-label budgets, with average error reduction of up to 41% across baselines. These results establish AdaKerNet as an effective approach for prediction from frozen multimodal representations in the scarce label regime. Additional structural ablations highlight the complementary contributions of AdaKerNet's components.

Figures & tables

Method	$N{=}100$	$200$	$300$	$500$	$1000$	$N{=}100$	$200$	$300$	$500$	$1000$
			PAD-UFES-20-AGE / BLIP-2					PAD-UFES-20-AGE / LLaVA
MLP	0.2586	0.2841	0.3359	0.3729	0.4129	0.2615	0.2925	0.3399	0.3511	0.4082
Transformer	0.2121	0.2409	0.3009	0.3367	0.3757	0.1883	0.1989	0.2648	0.2729	0.3701
AE $+$ linear	-2.7412	-0.7753	-0.1098	0.1991	0.3462	-2.6360	-0.8492	-0.0904	0.1722	0.3371
KRR	0.1338	0.2386	0.2898	0.3467	0.4120	-0.1548	0.0680	0.1569	0.2577	0.3521
SNGP	0.1100	0.0894	0.1997	0.2601	0.3325	0.0956	0.1409	0.1412	0.1887	0.2827

Table 1: Comparison of AdaKerNet with five baselines operating directly on raw MLLM representations across six continuous prediction benchmarks, varying label budgets

N

, and different MLLMs spanning BLIP-2, LLaVA-1.5, Qwen2.5-VL, and Gemini Embedding 2. Performance is measured by mean test

R^{2}

, with the best result in bold and the second best underlined . Avg. err. red. denotes AdaKerNet’s relative reduction in test MSE, computed as

100(R^{2}_{\mathrm{ours}}-R^{2}_{\mathrm{base}})/(1-R^{2}_{\mathrm{base}})

and averaged over the five baselines.

Dataset	Modalities	Target	Labeled pool	Test	$\sigma_{y}$
PAD-UFES-20-AGE	image, text, tabular	patient age	1,612	367	15.8
PAD-UFES-20-MOLE-LESION	image, text, tabular	$\log$ lesion diameter	1,043	284	0.579
AMAZON FASHION	image, text, tabular	$\log$ price	3,486	747	0.814
QUECHUA-VALENCE	audio, transcript	valence	2,800	600	0.207
SPEECHOCEAN762-FLUENCY	audio, transcript	fluency	2,000	2,500	1.492
SPEECHOCEAN762-PROSODIC	audio, transcript	prosodic	2,000	2,500	1.469

Table 2: Multimodal regression benchmarks.

\sigma_{y}

is the standard deviation of the target;

R^{2}

is scale-free and therefore comparable across datasets, while absolute errors are not.

PAD-UFES-20-AGE / BLIP-2
Method	$N{=}100$	$N{=}200$	$N{=}300$	$N{=}500$	$N{=}1000$
MLP on ${\bm{e}}$	0.2586	0.2841	0.3359	0.3729	0.4129
MLP on ${\bm{z}}$	0.2755	0.3133	0.3494	0.3717	0.4168
Err. red.	$+$ 2.3%	$+$ 4.1%	$+$ 2.0%	$-$ 0.2%	$+$ 0.7%
KRR on ${\bm{e}}$	0.1338	0.2386	0.2898	0.3467	0.4120
KRR with $k_{0}$ on ${\bm{z}}$	0.2531	0.3036	0.3414	0.3776	0.4245

Table 3: Effect of the Lipschitz-controlled features

{\bm{z}}

utilizing the test

R^{2}

metric. For each MLP or KRR predictor, the first row uses the raw MLLM representations

{\bm{e}}

and the second the Lipschitz-controlled features

{\bm{z}}

: MLP is the multilayer perceptron head of B-I, KRR on

{\bm{e}}

is B-IV, and KRR with

k_{0}

applies kernel ridge regression to

{\bm{z}}

with the reference kernel

k_{0}

, which for these three benchmarks has the same kernel form (RBF) as used by B-IV. Each Err. red. row gives the relative reduction (%) in test MSE obtained by replacing

{\bm{e}}

with

{\bm{z}}

(R^{2}_{{\bm{z}}}-R^{2}_{{\bm{e}}})/(1-R^{2}_{{\bm{e}}})

Method	$N{=}100$	$200$	$300$	$500$	$1000$	$N{=}100$	$200$	$300$	$500$	$1000$
			PAD-UFES-20-AGE / BLIP-2					PAD-UFES-20-AGE / LLaVA
MLP on ${\bm{z}}$	0.2755	0.3133	0.3494	0.3717	0.4168	0.2996	0.3484	0.3847	0.3897	0.4436
AdaKerNet	0.2750	0.3251	0.3579	0.3891	0.4261	0.3045	0.3588	0.3958	0.4061	0.4472
Rel. gain	$-$ 0.2%	$+$ 3.8%	$+$ 2.4%	$+$ 4.7%	$+$ 2.2%	$+$ 1.6%	$+$ 3.0%	$+$ 2.9%	$+$ 4.2%	$+$ 0.8%
			PAD-UFES-20-AGE / Qwen					PAD-UFES-20-MOLE-LESION / BLIP-2
MLP on ${\bm{z}}$	0.2740	0.3196	0.3474	0.3545	0.3812	0.1618	0.2205	0.2636	0.3322	0.3630

Table 4: Significance of joint task-adaptive neural kernel decoding on top of the Lipschitz-controlled features

{\bm{z}}

assessed using the test

R^{2}

metric. MLP on

{\bm{z}}

applies a multilayer perceptron head to

{\bm{z}}

; AdaKerNet is the full end-to-end model. The Rel. gain row gives AdaKerNet’s relative

R^{2}

gain (%) over MLP on

{\bm{z}}

(R^{2}_{\mathrm{AdaKerNet}}-R^{2}_{\mathrm{MLP}})/R^{2}_{\mathrm{MLP}}

Method	$N{=}100$	$200$	$300$	$500$	$1000$	$N{=}100$	$200$	$300$	$500$	$1000$
			PAD-UFES-20-AGE / BLIP-2					PAD-UFES-20-AGE / LLaVA
Deep MLP (AdaKerNet - $\mathcal{L}_{\mathrm{ker}}=0$ ) on ${\bm{z}}$	0.2714	0.3129	0.3463	0.3720	0.4158	0.2998	0.3492	0.3869	0.3903	0.4434
AdaKerNet	0.2750	0.3251	0.3579	0.3891	0.4261	0.3045	0.3588	0.3958	0.4061	0.4472
Rel. gain	$+$ 1.3%	$+$ 3.9%	$+$ 3.3%	$+$ 4.6%	$+$ 2.5%	$+$ 1.5%	$+$ 2.8%	$+$ 2.3%	$+$ 4.0%	$+$ 0.9%
			PAD-UFES-20-AGE / Qwen					PAD-UFES-20-MOLE-LESION / BLIP-2
Deep MLP (AdaKerNet - $\mathcal{L}_{\mathrm{ker}}=0$ ) on ${\bm{z}}$	0.2700	0.3226	0.3483	0.3546	0.3820	0.1598	0.2215	0.2661	0.3344	0.3653

Table 5: Significance of joint task-adaptive neural kernel decoding on top of the Lipschitz-controlled features

{\bm{z}}

assessed using the test

R^{2}

metric. MLP on

{\bm{z}}

now applies a (deeper) multilayer perceptron head (corresponding to AdaKerNet with

\mathcal{L}_{\mathrm{ker}}=0

) to

{\bm{z}}

; AdaKerNet is the full end-to-end model. The Rel. gain row gives AdaKerNet’s relative

R^{2}

gain (%) over MLP on

{\bm{z}}

(R^{2}_{\mathrm{AdaKerNet}}-R^{2}_{\mathrm{MLP}})/R^{2}_{\mathrm{MLP}}

Encoder	Kernel	Predictor	$N=100$	$N=200$	$N=300$	$N=500$	$N=1000$
BLIP-2	Matérn- $1/2$	$\widehat{k}_{\psi}$	0.2571	0.3132	0.3573	0.3917	0.4244
		$k_{0}$	0.2545	0.3025	0.3452	0.3779	0.4255
		Rel. gain	+1.01%	+3.53%	+3.49%	+3.66%	-0.27%
	Matérn- $3/2$	$\widehat{k}_{\psi}$	0.2592	0.3152	0.3571	0.3878	0.4279
		$k_{0}$	0.2524	0.3041	0.3449	0.3787	0.4256
		Rel. gain	+2.68%	+3.63%	+3.54%	+2.42%	+0.55%

Table 6: PAD-UFES-20-AGE benchmark. Kernel ridge regression (KRR) with the learned kernel

\widehat{k}_{\psi}

against KRR with the fixed reference kernel

k_{0}

based on the test

R^{2}

metric. The two share the same Lipschitz-controlled feature vector

{\bm{z}}

and the same reference kernel, and both select the ridge parameter based on the

N

labeled points. The Rel. gain (%) row is the relative

R^{2}

gain of KRR with

\hat{k}_{\psi}

over KRR with

k_{0}

, as

\,(R^{2}_{\hat{k}_{\psi}}-R^{2}_{k_{0}})/R^{2}_{\mathrm{k_{0}}}

Encoder	Kernel	Predictor	$N=100$	$N=200$	$N=300$	$N=500$	$N=1043$
BLIP-2	Matérn- $1/2$	$\widehat{k}_{\psi}$	0.1818	0.2241	0.2718	0.3354	0.3723
		$k_{0}$	0.1447	0.2167	0.2614	0.3303	0.3765
		Rel. gain	+25.57%	+3.43%	+3.97%	+1.55%	-1.13%
	Matérn- $3/2$	$\widehat{k}_{\psi}$	0.1830	0.2280	0.2708	0.3356	0.3719
		$k_{0}$	0.1433	0.2151	0.2604	0.3291	0.3713
		Rel. gain	+27.71%	+6.02%	+4.03%	+1.96%	+0.14%

Table 7: PAD-UFES-20-MOLE-LESION benchmark. Kernel ridge regression (KRR) with the learned kernel

\widehat{k}_{\psi}

against KRR with the fixed reference kernel

k_{0}

based on the test

R^{2}

metric. The two share the same Lipschitz-controlled feature vector

{\bm{z}}

and the same reference kernel, and both select the ridge parameter based on the

N

labeled points. The Rel. gain (%) row is the relative

R^{2}

gain of KRR with

\hat{k}_{\psi}

over KRR with

k_{0}

, as

\,(R^{2}_{\hat{k}_{\psi}}-R^{2}_{k_{0}})/R^{2}_{\mathrm{k_{0}}}

Encoder	Kernel	Predictor	$N=100$	$N=200$	$N=300$	$N=500$	$N=1000$
BLIP-2	Matérn- $1/2$	$\widehat{k}_{\psi}$	0.1948	0.3042	0.3834	0.4342	0.4890
		$k_{0}$	0.1068	0.2687	0.3774	0.4232	0.4863
		Rel. gain	+82.31%	+13.22%	+1.60%	+2.60%	+0.56%
	Matérn- $3/2$	$\widehat{k}_{\psi}$	0.1838	0.3033	0.3842	0.4334	0.4872
		$k_{0}$	0.0983	0.2693	0.3805	0.4274	0.4880
		Rel. gain	+86.92%	+12.64%	+0.98%	+1.41%	-0.17%

Table 8: AMAZON FASHION benchmark. Kernel ridge regression (KRR) with the learned kernel

\widehat{k}_{\psi}

against KRR with the fixed reference kernel

k_{0}

based on the test

R^{2}

metric. The two share the same Lipschitz-controlled feature vector

{\bm{z}}

and the same reference kernel, and both select the ridge parameter based on the

N

labeled points. The Rel. gain (%) row is the relative

R^{2}

gain of KRR with

\hat{k}_{\psi}

over KRR with

k_{0}

, as

\,(R^{2}_{\hat{k}_{\psi}}-R^{2}_{k_{0}})/R^{2}_{\mathrm{k_{0}}}

Encoder	Kernel	Predictor	$N=100$	$N=200$	$N=300$	$N=500$	$N=1000$
Gemini 2	Matérn- $1/2$	$\widehat{k}_{\psi}$	0.0675	0.1758	0.2099	0.2331	0.2717
		$k_{0}$	0.0229	0.1620	0.2006	0.2267	0.2697
		Rel. gain	+194.89%	+8.48%	+4.60%	+2.82%	+0.72%
	Matérn- $3/2$	$\widehat{k}_{\psi}$	0.0596	0.1753	0.2077	0.2336	0.2717
		$k_{0}$	0.0184	0.1618	0.2028	0.2275	0.2689
		Rel. gain	+223.91%	+8.34%	+2.41%	+2.68%	+1.03%

Table 9: QUECHUA-VALENCE benchmark. Kernel ridge regression (KRR) with the learned kernel

\widehat{k}_{\psi}

against KRR with the fixed reference kernel

k_{0}

based on the test

R^{2}

metric. The two share the same Lipschitz-controlled feature vector

{\bm{z}}

and the same reference kernel, and both select the ridge parameter based on the

N

labeled points. The Rel. gain (%) row is the relative

R^{2}

gain of KRR with

\hat{k}_{\psi}

over KRR with

k_{0}

, as

\,(R^{2}_{\hat{k}_{\psi}}-R^{2}_{k_{0}})/R^{2}_{\mathrm{k_{0}}}

Encoder	Kernel	Predictor	$N=100$	$N=200$	$N=300$	$N=500$	$N=1000$
Gemini 2	Matérn- $1/2$	$\widehat{k}_{\psi}$	0.2032	0.3310	0.3536	0.3901	0.4170
		$k_{0}$	0.1603	0.3123	0.3602	0.3747	0.4131
		Rel. gain	+26.78%	+5.97%	-1.81%	+4.11%	+0.94%
	Matérn- $3/2$	$\widehat{k}_{\psi}$	0.2076	0.3314	0.3558	0.3915	0.4177
		$k_{0}$	0.1519	0.3105	0.3620	0.3794	0.4152
		Rel. gain	+36.65%	+6.72%	-1.71%	+3.18%	+0.60%

Table 10: SPEECHOCEAN762-FLUENCY benchmark. Kernel ridge regression (KRR) with the learned kernel

\widehat{k}_{\psi}

against KRR with the fixed reference kernel

k_{0}

based on the test

R^{2}

metric. The two share the same Lipschitz-controlled feature vector

{\bm{z}}

and the same reference kernel, and both select the ridge parameter based on the

N

labeled points. The Rel. gain (%) row is the relative

R^{2}

gain of KRR with

\hat{k}_{\psi}

over KRR with

k_{0}

, as

\,(R^{2}_{\hat{k}_{\psi}}-R^{2}_{k_{0}})/R^{2}_{\mathrm{k_{0}}}

Encoder	Kernel	Predictor	$N=100$	$N=200$	$N=300$	$N=500$	$N=1000$
Gemini 2	Matérn- $1/2$	$\widehat{k}_{\psi}$	0.2165	0.3423	0.3487	0.3809	0.4128
		$k_{0}$	0.1868	0.3120	0.3463	0.3622	0.4077
		Rel. gain	+15.95%	+9.70%	+0.69%	+5.16%	+1.23%
	Matérn- $3/2$	$\widehat{k}_{\psi}$	0.2099	0.3407	0.3503	0.3825	0.4127
		$k_{0}$	0.1783	0.3103	0.3482	0.3677	0.4099
		Rel. gain	+17.71%	+9.81%	+0.61%	+4.02%	+0.69%

Table 11: SPEECHOCEAN762-PROSODIC benchmark. Kernel ridge regression (KRR) with the learned kernel

\widehat{k}_{\psi}

against KRR with the fixed reference kernel

k_{0}

based on the test

R^{2}

metric. The two share the same Lipschitz-controlled feature vector

{\bm{z}}

and the same reference kernel, and both select the ridge parameter based on the

N

labeled points. The Rel. gain (%) row is the relative

R^{2}

gain of KRR with

\hat{k}_{\psi}

over KRR with

k_{0}

, as

\,(R^{2}_{\hat{k}_{\psi}}-R^{2}_{k_{0}})/R^{2}_{\mathrm{k_{0}}}

BLIP-2
Method	$N=100$	$N=200$	$N=300$	$N=500$	$N=1000$
AdaKerNet, one row per reference kernel $k_{0}$
AdaKerNet (Matérn- $1/2$ )	0.2753	0.3226	0.3562	0.3930	0.4237
AdaKerNet (Matérn- $3/2$ )	0.2785	0.3247	0.3579	0.3899	0.4260
AdaKerNet (Matérn- $5/2$ )	0.2785	0.3229	0.3567	0.3903	0.4256
AdaKerNet (RBF)	0.2750	0.3251	0.3579	0.3891	0.4261

Table 12: Q1 — PAD-UFES-20-AGE benchmark. AdaKerNet using one of four distinct reference kernels, against the baselines on raw representations based on test

R^{2}

. Best per column in bold , second best underlined .

Gemini 2
Method	$N=100$	$N=200$	$N=300$	$N=500$	$N=1000$
AdaKerNet, one row per reference kernel $k_{0}$
AdaKerNet (Matérn- $1/2$ )	0.2168	0.3328	0.3519	0.3917	0.4194
AdaKerNet (Matérn- $3/2$ )	0.2130	0.3316	0.3543	0.3946	0.4198
AdaKerNet (Matérn- $5/2$ )	0.2056	0.3307	0.3564	0.3960	0.4199
AdaKerNet (RBF)	0.2033	0.3321	0.3568	0.3963	0.4204

Table 13: Q1 — SPEECHOCEAN762-FLUENCY benchmark. AdaKerNet using one of four distinct reference kernels, against the baselines on raw representations based on test

R^{2}

. Best per column in bold , second best underlined .

	BLIP-2	LLaVA	Qwen
Method	$N=1612$	$N=1612$	$N=1612$
AdaKerNet	0.4365	0.4720	0.4106
Baselines on raw representations ${\bm{e}}$
MLP	0.4479	0.4498	0.1642
Transformer	0.4054	0.4187	0.2243
Autoencoder $+$ linear	0.3870	0.3816	0.3209

Table 14: Q2 — PAD-UFES-20-AGE benchmark in the less sparse regime. AdaKerNet against the baselines on raw representations at every

N>1000

AdaKerNet uses the RBF reference kernel reported in the main results. Best per column in bold , second best underlined .

	BLIP-2	LLaVA	Qwen
Method	$N=1043$	$N=1043$	$N=1043$
AdaKerNet	0.3716	0.3246	0.2724
Baselines on raw representations ${\bm{e}}$
MLP	0.1856	0.2134	0.0654
Transformer	0.2678	0.1260	0.1051
Autoencoder $+$ linear	0.3330	0.2478	0.1755

Table 15: Q2 — PAD-UFES-20-MOLE-LESION in the less sparse regime. AdaKerNet against the baselines on raw representations at every

N>1000

. AdaKerNet uses the RBF reference kernel reported in the main results. Best per column in bold , second best underlined .

Baselines on raw representations ${\bm{e}}$
	BLIP-2		LLaVA		Qwen
Method	$N=2000$	$N=3486$	$N=2000$	$N=3486$	$N=2000$	$N=3486$
AdaKerNet	0.5458	0.5792	0.4648	0.5019	0.5321	0.5608
MLP	0.4860	0.5285	0.3132	0.3184	0.4307	0.4840
Transformer	0.4643	0.5108	0.3902	0.4202	0.4596	0.4993
Autoencoder $+$ linear	0.3388	0.3602	0.3154	0.3386	0.4096	0.4291

Table 16: Q2 — AMAZON FASHION benchmark in the less sparse regime. AdaKerNet against the baselines on raw representations at every

N>1000

. AdaKerNet uses the Matérn-

1/2

reference kernel reported in the main results. Best per column in bold , second best underlined .

Method	$N=2000$	$N=2800$
AdaKerNet	0.3111	0.3260
Baselines on raw representations ${\bm{e}}$
MLP	0.1416	0.2035
Transformer	0.1932	0.2167
Autoencoder $+$ linear	0.2520	0.2631
KRR	0.3236	0.3489

Table 17: Q2 — QUECHUA-VALENCE benchmark in the less sparse regime. AdaKerNet against the baselines on raw representations at every

N>1000

. AdaKerNet uses the Matérn-

5/2

reference kernel reported in the main results. Best per column in bold , second best underlined .

Method	$N=2000$
AdaKerNet	0.4399
Baselines on raw representations ${\bm{e}}$
MLP	0.4079
Transformer	0.3722
Autoencoder $+$ linear	0.3454
KRR	0.4299

Table 18: Q2 — SPEECHOCEAN762-FLUENCY benchmark the less sparse regime. AdaKerNet against the baselines on raw representations at every

N>1000

. AdaKerNet uses the RBF reference kernel reported in the main results. Best per column in bold , second best underlined .

Method	$N=2000$
AdaKerNet	0.4327
Baselines on raw representations ${\bm{e}}$
MLP	0.4085
Transformer	0.3850
Autoencoder $+$ linear	0.3622
KRR	0.4485

Table 19: Q2 — SPEECHOCEAN762-PROSODIC benchmark in the less sparse regime. AdaKerNet against the baselines on raw representations at every

N>1000

. AdaKerNet uses the Matérn-

3/2

reference kernel reported in the main results. Best per column in bold , second best underlined .

QUECHUA-VALENCE
Reference kernel	$N=200$	$N=300$	$N=500$
$\lambda=0.0001$
Matérn- $1/2$	0.1721	0.2086	0.2297
Matérn- $3/2$	0.1720	0.2079	0.2320
Matérn- $5/2$	0.1705	0.2089	0.2330
RBF	0.1735	0.2087	0.2341

Table 20: Q3 — AdaKerNet on the QUECHUA-VALENCE benchmark, Gemini 2 representation, evaluated for different values of

\lambda

, at each of four reference kernels based on the test

R^{2}

. For

\lambda\in\{0.1,1,10\}

the value selected for that kernel and

N

is given in parentheses. Best per column in bold .

Method	$N{=}100$	$200$	$300$	$500$	$1000$	$N{=}100$	$200$	$300$	$500$	$1000$
			PAD-UFES-20-AGE / BLIP-2					PAD-UFES-20-AGE / LLaVA
KRR (Matérn- $1/2$ ) on ${\bm{e}}$	0.1263	0.2504	0.3025	0.3591	0.4168	-0.0746	0.1445	0.2320	0.3244	0.4093
AdaKerNet	0.2750	0.3251	0.3579	0.3891	0.4261	0.3045	0.3588	0.3958	0.4061	0.4472
Err. red.	$+$ 17.0%	$+$ 10.0%	$+$ 7.9%	$+$ 4.7%	$+$ 1.6%	$+$ 35.3%	$+$ 25.0%	$+$ 21.3%	$+$ 12.1%	$+$ 6.4%
			PAD-UFES-20-AGE / Qwen					PAD-UFES-20-MOLE-LESION / BLIP-2
KRR (Matérn- $1/2$ ) on ${\bm{e}}$	-0.0611	0.1121	0.1909	0.2813	0.3675	-0.0323	0.1082	0.1667	0.2624	0.3380

Table 21: A daKerNet against kernel ridge regression on the raw representation

{\bm{e}}

with a Matérn-

1/2

kernel instead of the RBF kernel used in the main results, on all six benchmarks based on test

R^{2}

metric. AdaKerNet uses the reference kernel reported for each benchmark in the main results: RBF for both PAD-UFES-20 targets and for SPEECHOCEAN762-FLUENCY, Matérn-

1/2

for AMAZON-FASHION, Matérn-

5/2

for QUECHUA-VALENCE and Matérn-

3/2

for SPEECHOCEAN762-PROSODIC. The higher entry at each

N

is in bold . Err. red. is AdaKerNet’s relative reduction (%) in test MSE over the KRR baseline as

(R^{2}_{\text{ours}}-R^{2}_{\text{KRR}})/(1-R^{2}_{\text{KRR}})

Method	$N{=}100$	$200$	$300$	$500$	$1000$	$N{=}100$	$200$	$300$	$500$	$1000$
			PAD-UFES-20-MOLE-LESION / BLIP-2					PAD-UFES-20-MOLE-LESION / LLaVA
TRF on ${\bm{z}}$	0.1513	0.2255	0.2713	0.3395	0.3815	0.1252	0.2029	0.2413	0.2865	0.3143
AdaKerNet	0.1777	0.2291	0.2731	0.3347	0.3716	0.1475	0.2134	0.2497	0.2925	0.3246
Rel. gain	$+$ 17.4%	$+$ 1.6%	$+$ 0.6%	$-$ 1.4%	$-$ 2.6%	$+$ 17.8%	$+$ 5.2%	$+$ 3.4%	$+$ 2.1%	$+$ 3.3%
			PAD-UFES-20-MOLE-LESION / Qwen					QUECHUA-VALENCE / Gemini 2
TRF on ${\bm{z}}$	0.1100	0.1679	0.1935	0.2331	0.2760	0.0261	0.1737	0.2073	0.2338	0.2753

Table 22: Task-adaptive kernel deformation stage of AdaKerNet against spectral kernel learning on the same Lipschitz-controlled features

{\bm{z}}

, assessed using the test

R^{2}

metric. TRF on

{\bm{z}}

is Tuned Random Features ( Shilton et al., 2022 ) : frequencies are drawn once from the reference density and a per-frequency density ratio is learned alongside the weights. AdaKerNet instead deforms the reference kernel through a learned feature map, which need not remain translation-invariant. On each benchmark both are reported at that benchmark’s own reference kernel, which is the density TRF samples its frequencies from, so neither is given a choice the other lacks. The Rel. gain row gives AdaKerNet’s relative

R^{2}

gain (%) over TRF as

(R^{2}_{\mathrm{AdaKerNet}}-R^{2}_{\mathrm{TRF}})/R^{2}_{\mathrm{TRF}}

Explore similar work

CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks
Apr 1, 2026Wish Suharitdamrong, Tony Alex, Muhammad Awais +1Vision-Language Model Adaptation Cross-Modal
Surrey Institute for People-Centred AI, University of Surrey, Guildford, GU2 7XH, UK · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK
MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scale
Aug 11, 2026Yuhang Yao, Zeyu Wang, Wanyi Chen +8Large Language Model Adaptation Low-Rank Adaptation Adapters
Carnegie Mellon University · University of California, Los Angeles · Soochow University +3
Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach
May 20, 2026Herman Bergström, Aditya Mehrotra, Rahul G. KrishnanMultimodal Classification Multimodal Learning
Chalmers University of Technology and University of Gothenburg · Vector Institute · University of Toronto

Apr 1, 2026, cs.CVJ/K move · Enter open · S save

CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks

Wish Suharitdamrong, Tony Alex, Muhammad Awais, Sara Atito

Surrey Institute for People-Centred AI, University of Surrey, Guildford, GU2 7XH, UK · Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK

Vision-Language Model Adaptation Cross-Modal Unimodal Encoders Downstream Reasoning

Open paper AskPDF arXiv

Foundation models have revolutionized AI, but adapting them efficiently for multimodal tasks, particularly in dual-stream architectures composed of unimodal encoders, such as DINO and BERT, remains a significant challenge. ParameterEfficient Fine-Tuning (PEFT) methods like LowRank Adaptation (LoRA) enable lightweight adaptation, yet they operate in isolation within each modality, limiting their ability in capturing cross-modal interactions. In this paper, we take a step in bridging this gap with Cross-Modal LowRank Adaptation (CoLA), a novel PEFT framework that extends LoRA by introducing a dedicated inter-modal adaptation pathway alongside the standard intra-modal one. This dual-path design enables CoLA to adapt unimodal foundation models to multimodal tasks effectively, without interference between modality-specific and crossmodal learning. We evaluate CoLA across a range of vision-language (RefCOCO, RefCOCO+, RefCOCOg) and audio-visual (AVE, AVS) benchmarks, where it consistently outperforms LORA, achieving a relative gain of around 3% and 2%, respectively, while maintaining parameter efficiency. Notably, CoLA enables the first multitask PEFT framework for visual grounding, bridging a key gap in efficient multimodal adaptation. Code is available at https://github.com/peterwisu/CoLA

Abstract

Explore similar work

CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks

MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scale

Modular Multimodal Classification Without Fine-Tuning: A Simple Compositional Approach