Joint, Decoupled, or Agent: When Each Multimodal Alignment Paradigm Wins

Yi Fang; Gaurav Srivastava; Sirui Ding; Xuan Wang

cs.AIOct 16, 2026

Joint, Decoupled, or Agent: When Each Multimodal Alignment Paradigm Wins

Authors: Yi Fang, Gaurav Srivastava, Sirui Ding, Xuan Wang

Organizations: Department of Computer Science, Virginia Tech, Blacksburg, VA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA

Abstract

Jointly pretrained multimodal encoders are a common way to combine modalities, while alternatives integrate their information during downstream training or inference. When does each approach win? We compare three paradigms, instantiated with public checkpoints, across 16 paired datasets in five domains and multiple training-label budgets. JOINT trains a fusion head on frozen, jointly pretrained encoders. DECOUPLED trains a fusion head on frozen, independently pretrained encoders. AGENT gives a frozen LLM task instructions, labelled examples, available sample text, and supervised unimodal readouts. No paradigm dominates the complete 144 dataset-budget combinations: DECOUPLED wins 54, JOINT 44, and AGENT 46 by mean utility. ScienceQA, MELD, and MetMeme shift from AGENT to fusion as supervision increases. Between the fusion paradigms, JOINT leads on four of five QA datasets across all evaluated budgets, while higher unimodal 5-NN accuracy is associated with a relative shift toward DECOUPLED across eight classification datasets (Spearman $ρ=-0.67$ ). We test whether representation and task diagnostics predict these preferences on held-out datasets before fitting their candidate systems. A two-step cascade combines these diagnostics with task and budget metadata to select AGENT or fusion, then JOINT or DECOUPLED. Under leave-one-group-out evaluation, it selects the winner in 71.4% of cells versus 54.2% for metadata-only routing, with mean relative regret 0.028 versus 0.057. Using geometry, supervised representation diagnostics, and text-label matching together yields higher accuracy and lower regret than using any one of these sources with metadata. These results connect paradigm preferences to supervision budget, task format, and unimodal 5-NN accuracy. Target-task diagnostics thus support selecting among the three paradigms before candidate fitting.

Figures & tables

Figure 1: Paradigm wins across

144

dataset–budget combinations. The system with the highest mean utility wins.

Figure 2: Persistent and changing paradigm preferences across supervision budgets. Utility versus labelled examples (log scale). Lines mark main-table budgets. More datasets: Figure 6 .

Figure 3: QA input controls. Full-system minus control accuracy, in percentage points. Each contrast compares the full system with a separate input control on matched full-budget test examples. Table 13 lists all controls and the input retained in each condition.

Figure 4: Diagnostics and fusion preference. A: QA win shares over

n

budgets. B: Unimodal

5

-NN accuracy versus normalized Joint–Decoupled margin across classification datasets. Dashed line: descriptive fit. Definitions and sensitivity checks: Appendix A.13 .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Win map of the

48

main-table cells , coloured and lettered by the winning paradigm.

\dagger

marks the four cells with a Joint – Decoupled gap of at most

0.005

. Rows are grouped by task type, which use different budgets, and ordered by domain. The bottom strip counts winners per budget tier.

Figure 6: Learning curves for six datasets over each dataset’s full budget grid (log scale). Vertical lines mark the budgets of the main tables. Teal = Joint , coral = Decoupled , ochre = Agent . The curves distinguish persistent preferences from budget-dependent reversals. The budget dependence of Agent includes changes in its fitted readouts and available demonstrations.

Dataset	Budget	Joint	Decoupled	Gap	Comb. std	Status
Classification (utility $=$ accuracy)
CUB	20-shot	$0.811\pm 0.004$	$\mathbf{0.899}\pm 0.002$	$-0.088$	$0.005$	clear
	10%	$0.641\pm 0.013$	$\mathbf{0.830}\pm 0.009$	$-0.189$	$0.016$	clear
	100%	$0.833\pm 0.004$	$\mathbf{0.917}\pm 0.001$	$-0.084$	$0.004$	clear
CrisisMMD	20-shot	$\mathbf{0.458}\pm 0.032$	$0.306\pm 0.077$	$+0.153$	$0.083$	clear
	10%	$\mathbf{0.634}\pm 0.008$	$0.476\pm 0.011$	$+0.158$	$0.013$	clear

Appendix

Table 4: Per-seed variance of Table 1 . Score columns report mean

\pm

std over

5

seeds: accuracy for CLS and positive MAE for REG. Gap is

U_{J}-U_{D}

, with utility equal to accuracy or

-

MAE. A positive gap favours Joint . Bold marks the better of Joint and Decoupled , ties going to Decoupled . Combined std

=\sqrt{s_{J}^{2}+s_{D}^{2}}

. A cell is a near-tie if

|\mathrm{gap}|\leq 0.005

, noisy if

|\mathrm{gap}|

is below the combined std, and clear otherwise.

Classification
Dataset	Budget	Joint wins	Decoupled wins	Agent wins
CUB	20-shot	0	5	0
	10%	0	2	3
	100%	0	5	0
CrisisMMD	20-shot	2	0	3
	10%	3.5	0	1.5

Appendix

Table 5: Five-seed winner stability for the

48

main-table cells. Each row counts wins by Joint , Decoupled , and Agent over five corresponding seeds. A sole winner receives one win. Tied best configurations share one win equally, so every row sums to five. Higher accuracy wins for classification/QA and lower MAE for regression. Counts use the supplied three-decimal seed scores, so ties mean equality at that precision, not statistical equivalence. Bold marks the largest count in a row.

Figure 7: Diagnostic test-best fusion head per cell , coloured by the head with the best five-seed mean on the default pair and labelled with the spread between the best and the worst of the eight heads (task metric units). The spread is widest on the MCQ tasks and at the smallest budgets, and at most

0.13

on classification at full budget.

Figure 8: Diagnostic test-performance rank of fusion heads across

16

datasets at

100\%

supervision.

\bigstar

= paradigm best.

Figure 9: Backbone scaling ablation. Left: per-dataset scaling curves at full budget for the CLIP/SigLIP/EVA02 family. Solid lines = Joint , dashed lines = Decoupled . Right: the Joint minus Decoupled gap across scale tiers (base/large/huge). Datasets above the zero line favour Joint and datasets below favour Decoupled . The gap keeps its sign from base to large on both datasets. At the huge tier the only matched unaligned tower is MAE-ViT-H/14 (Table 8 ).

Figure 10: General and domain-specific encoder configurations. (a) Fracture, Joint with the general CLIP-B/16 and three biomedical paired encoders: PubMedCLIP exceeds the default Decoupled pair at the largest budgets, while also differing in visual patch size and text architecture. (b) Fracture, Decoupled with domain-specific vision towers: none beats the general DINOv2. (c) MELD, general and music CLAP for Joint and three audio encoders for Decoupled , which leads at every percentage budget.

Figure 11: In-context examples (

N_{\mathrm{ICL}}{=}10

vs.

0

). Per-dataset Agent accuracy at the three label budgets. In-context examples help most on MetMeme and CrisisMMD and cost up to

0.057

on OxfordPet. Numerical values in Table 10 .

Figure 12: LLM backbone for the deployed Agent. Per-dataset Agent utility under Qwen3-8B, Qwen3-32B (deployed), Gemma-3-27B-it, and GPT-OSS-120B at the reported label budgets. At

20

-shot Qwen3-8B is below Qwen3-32B on

7

of the

8

classification datasets. Numerical values in Tables 11 and 12 .

Figure 13: Tool-dispatch mode (Inline-1 vs. Multi-2). Per-dataset accuracy under the two dispatch modes on the seven classification datasets where both readouts were available to the Multi-2 controller. Multi-2 is usually less accurate, most visibly on CUB and OxfordPet, because the controller often calls the uninformative readout. Inline-1, deployed throughout, uses one LLM turn instead of two.

Dataset	Budget	Non-text	Text	Avg.	Agent	Joint	Decoupled
CrisisMMD	20-shot	0.235	0.090	0.221	0.460	0.458	0.306
CrisisMMD	100%	0.500	0.467	0.510	0.683	0.668	0.538
Food101	20-shot	0.695	0.074	0.695	0.872	0.767	0.695
Food101	10%	0.736	0.074	0.737	0.880	0.804	0.733
Food101	100%	0.802	0.122	0.810	0.906	0.862	0.808
MELD	20-shot	0.309	0.281	0.342	0.591	0.375	0.445

Appendix

Table 14: Simple readout and LLM-only comparisons with Agent on CrisisMMD (

20

-shot and

100\%

), Food101 (all three budgets), MELD (

20

-shot), and MetMeme (

20

-shot), where Agent wins in Table 1 (accuracy). Non-text and Text are the two single-modality readouts Agent receives, and Avg. averages their class probabilities with no trained parameter. Readouts and average are the mean over five label-subsampling seeds, on the same complete test split as Agent . Food101 stores the ten highest logits of each readout, and the remaining classes take the tenth. The lower panel compares Food101 with LLM-only, which retains raw-text access and omits both readouts as in Table 13 . Differences are five-seed mean contrasts in percentage points.

Configuration	Step 1 ACC	Step 2 ACC	Final ACC
Cascade, default features	0.806	0.843	0.714
Direct, default features	0.654	0.627	0.450
Cascade, base-tier general-domain features	0.804	0.867	0.718

Appendix

Table 17: Stage-level and final routing accuracy. Results use

\epsilon=0

15

dataset groups, and averages over five labelled-set seeds. Final accuracy covers all

144

dataset–budget combinations. For all three configurations, Step 2 accuracy is computed on the same subset whose true winner is Joint or Decoupled . The feature-encoder variant replaces the general-domain feature encoders with the base-tier configuration.

$\epsilon$	Accuracy difference [95% CI]	Relative-regret difference [95% CI]
0%	$+0.264\;[0.139,0.399]$	$-0.063\;[-0.113,-0.024]$
0.2%	$+0.265\;[0.099,0.452]$	$-0.072\;[-0.164,-0.016]$
0.5%	$+0.267\;[0.101,0.452]$	$-0.066\;[-0.159,-0.011]$
1%	$+0.024\;[-0.206,0.184]$	$-0.008\;[\text{--}]$
2%	$+0.078\;[-0.009,0.149]$	$-0.009\;[-0.024,0.002]$

Appendix

Table 18: Cascade minus direct three-way classifier. Positive accuracy differences and negative relative-regret differences favour the cascade. Differences use higher-precision summaries. Reference-tolerance accuracy is

102.8/144

. Values are rounded independently and may differ by

0.001

from subtraction of displayed means. Intervals are pointwise paired group-bootstrap percentiles (

15

groups,

4{,}000

draws, NumPy default_rng(0) ). The

1\%

regret difference uses displayed means. Its interval is not reported.

$\epsilon$	Accuracy difference [95% CI]	Relative-regret difference [95% CI]
0%	$+0.228\;[0.003,0.457]$	$-0.099\;[-0.199,-0.021]$
0.2%	$+0.197\;[-0.044,0.448]$	$-0.091\;[-0.193,-0.010]$
0.5%	$+0.201\;[-0.031,0.445]$	$-0.089\;[-0.191,-0.009]$
1%	$+0.116\;[-0.161,0.348]$	$-0.084\;[\text{--}]$
2%	$+0.133\;[-0.072,0.311]$	$-0.050\;[-0.106,-0.000812]$

Appendix

Table 19: Cascade minus TB-Majority. Positive accuracy differences and negative relative-regret differences favour the cascade. Differences use higher-precision summaries. Reference-tolerance accuracy is

102.8/144

. Values are rounded independently and may differ by

0.001

from subtraction of displayed means. Intervals are pointwise paired group-bootstrap percentiles (

15

groups,

4{,}000

draws, NumPy default_rng(0) ). The

1\%

regret difference uses displayed means. Its interval is not reported.

Dataset	$20$ -shot	$10\%$	$100\%$
PDBbind	$3.3\%$	$2.8\%$	$2.8\%$
KIBA	$0.3\%$	$1.0\%$	$2.9\%$
IEMOCAP	$0.2\%$	$0.1\%$	$0.2\%$

Appendix

Table 21: Unparseable outputs of the default AGENT on regression tasks. Entries are the observed failure percentages at the three main-table budgets. Unparseable outputs are replaced by numerical zero before MAE evaluation. These rates do not quantify the corresponding MAE contribution or the conditional error on successfully parsed outputs.

Label-free geometry ( $12$ features)
Extraction angle	#	Base functions	What it captures
Per-modality geometry	$10$	$\mathrm{unif}$ , $\mathrm{srank}$ , $\mathrm{decay}$ , $\mathrm{cond}$ , $\mathrm{iso}$	shape of each side’s embedding cloud
Cross-modal alignment	$2$	$\mathrm{CKA}$ , $\mathrm{mknn}_{10}$	agreement between the two spaces
Label, budget, and task features ( $14$ features)
ANOVA separability	$2$	$f_{\mathrm{F}}$	class separability of each side
Frozen-embedding task signal	$3$	$\mathrm{knn5}$	label signal a $k$ -NN recovers, per side and on $[A;B]$

Appendix

Table 28: The

26

base features of

\phi(d,K)

, grouped by extraction angle. Geometry and kNN features use

E_{a},E_{b}

. The text–label probe additionally uses a fixed sentence encoder. Each per-side base function is evaluated on both modalities and kept as its minimum and maximum over the two sides.

n_{\mathrm{train}}

is the size of the training split and

|\mathcal{Y}_{d}|

the number of classes (five quantile buckets for regression, undefined for MCQ and imputed). CKA and mutual-kNN require paired observations. Labelled kNN on

[A;B]

requires paired labelled observations. Fold-defined missingness indicators are appended before standardization. The two ANOVA and three labelled kNN features form the supervised representation diagnostics group used in the ablations. Per-feature definitions are in Appendix D.1 .

Feature group	Observations required	Labels required
Per-side uniformity, rank, decay, condition, isotropy	Samples from each modality, with no cross-modal correspondence	None
Per-side ANOVA and kNN score	Modality-specific samples, with no cross-modal correspondence	Compatible task labels on each side
CKA and mutual-kNN overlap	Corresponding observations of both modalities	None
kNN score on $[A;B]$	Paired observations	Labels for those paired observations
$\Delta_{\mathrm{zs}}$ (Step 1)	Text inputs and fixed sentence-encoder probe, with no cross-modal correspondence	Text-side labels and classification class names
Budget descriptors	Training-pool and labelled-set sizes	Task/class metadata, with a separate size convention if the two pools differ

Appendix

Table 29: Data required by routing features. The complete pipeline operates on paired target data and selects from these feature groups at each step. The table distinguishes features that use sample correspondence from those computed independently within each modality.

A. System inference and deployment
Item	Joint	Decoupled	Agent	LLM-only (reference)
Models run	CLIP-ViT-L/14 + head	DINOv2-L/14 + GPT-2 + head	Qwen3-32B + two readout pipelines	Qwen3-32B
Parameters	$0.43$ B	$0.43$ B	$\approx 32.43$ B	$32$ B
LLM tokens/sample	Not applicable	Not applicable	$1.0$ k– $8.5$ k	$1.0$ k assumption for estimate
FLOPs/sample [E]	$1.8\times 10^{11}$	$1.8\times 10^{11}$	$6.4\times 10^{13}$ to $5.4\times 10^{14}$	$\approx 6.4\times 10^{13}$ at assumed lower token count
FLOPs relative to fusion [E]	$1\times$	$1\times$	$\sim 360$ – $3{,}000\times$	$\sim 360\times$ at assumed lower token count

Appendix

Table 31: Computational costs and deployment requirements. A: per-sample inference and deployment for the general-domain default configurations. B: dataset-level preparation and evaluation on one A100-80GB. C: one-time routing diagnostics per cell. [M] denotes measured values, [E] estimates, and — an unreported measurement. The panels have different accounting units and must not be summed.

Accuracy gain threshold	Cells exceeding threshold	Share of $14$ positive-gain cells
$>0$ pp	$14/39$	$14/14$
$>1$ pp	$13/39$	$13/14$
$>2$ pp	$12/39$	$12/14$
$>5$ pp	$8/39$	$8/14$

Appendix

Table 32: AGENT gain over the better fusion configuration. Counts are computed from the rounded main-table mean accuracies on

39

classification/QA cells. Thresholds are strict and expressed in absolute percentage points (pp). They are descriptive, not significance tests or universal deployment requirements. The

9

regression cells use MAE and are excluded.

Relative advantage over the best FM	Number of cells
$<0.5\%$	$3$
$<1\%$	$4$
$<2\%$	$5$
$<5\%$	$8$

Appendix

Table 33: Small AGENT advantages across the full routing grid. Of

144

cells,

46

have strictly positive

\Delta=U_{A}-\max(U_{J},U_{D})

\epsilon=0

. The relative margin is

\Delta/|U^{\star}|

, using the same denominator as the tie tolerance. Here

U^{\star}=U_{A}

. Counts are cumulative within these

46

cells. Accuracy and negative MAE enter through utility, not pooled raw metric differences.

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 766 papers

May 31, 2026

Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning

The choice between cross-attention and concatenation for multimodal fusion remains governed by practitioner intuition rather than principled understanding. In this paper, we demonstrate that feature alignment quality, not data scale alone, is the primary determinant of which fusion strategy excels. Through controlled experiments on Flickr8k using two feature extraction backbones (ResNet18 and CLIP ViT-B/32), we show that concatenation outperforms cross-attention by 4.1-5.1 percentage points across all tested scales (2048-16384 samples) when features are pre-aligned by a vision-language pretraining objective. We provide a theoretical explanation grounded in sample complexity analysis: concatenation requires O(d_v + d_t) samples to learn its fusion projection, while cross-attention requires O(d_v * d_t) samples to learn bilinear attention weights, over 256 times as many for 512-dimensional CLIP features. When features are already aligned, the approximation error gap between the two methods vanishes, and concatenation's sample efficiency dominates at all practical dataset sizes. An alignment degradation study confirms a monotonic trend: as feature alignment degrades, concatenation's advantage grows from 1.3% to 2.8%. These findings provide a principled decision framework for fusion method selection in multimodal systems, with direct implications for the design of Multimodal Large Language Models.

Zhiqiang Zhou, Xuezhen Xie

Hunan Chemical Industry Vocational and Technical College

Cross-Attention Multimodal Fusion Feature Fusion Multimodal Understanding

Ask PDF

Jul 14, 2026

Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis

Multimodal fusion must simultaneously refine modality-specific signals and model cross-modal interactions; two competing objectives typically entangled within the same operation. We propose SeRIn (Segregate, Refine, Integrate), a multimodal LM fusion scheme that enforces this separation as an architectural prior. Modality-specific representations evolve along isolated pathways, each refined against its respective encoder context, while a dedicated cross-modal pathway accumulates their joint evolution without contaminating unimodal streams. Full cross-modal interaction is deferred to a final prediction step - ablations confirm that structured interactions, not added capacity, drive the gains; gate analysis under visual corruption reveals emergent modality reweighting without explicit supervision. SeRIn achieves state-of-the-art results on CH-SIMS and CMU-MOSEI, improving all metrics on both benchmarks.

Alexios Filippakopoulos, Elias Kallioras, Nikolaos Xiros +2

National Technical University of Athens, Greece · Athena Research Center, Greece · University of Bern, Switzerland +2

Cross-Modal Representation Learning Multimodal Large Language Models Multimodal Sentiment Analysis Multimodal Fusion Data Fusion

Ask PDF

May 20, 2026

Multimodal LLMs under Pairwise Modalities

Despite the impressive results achieved by multimodal large language models (MLLMs), their training typically relies on jointly curated multimodal data, requiring substantial human effort to construct multi-way aligned datasets and thereby limiting scalability across domains. In this work, we explore training MLLMs by only leveraging multiple paired modalities as a surrogate for the full joint multimodal distribution. Specifically, we first provide a theoretical analysis of the conditions under which the representations are identifiable with only observing pairwise modalities. Building on this analysis, we propose a representation learning framework for aligning latent representations across modalities using only pairwise data. The framework consists of two stages: latent representation alignment and cross-modal recomposition. Specifically, in the first stage, we learn the shared latent space across modalities by both self-modal reconstruction and pair-wise contrastive learning. We also incorporate an inductive bias in the contrastive learning process by partially aligning and minimal latent specification. In stage two, we integrate the encoder of newly introduced modalities with the decoders of the pre-trained modalities to facilitate cross-modal transfer and generation. We evaluate our method by newly adding 3D point clouds and tactile modalities into pre-trained MLLMs with three modality pairs and show that, by learning an aligned latent representation space, our model achieves strong cross-modal performance.

Yan Li, Yunlong Deng, Yuewen Sun +3

Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University

Cross-Modal Learning Representation Identifiability Cross-Modal Alignment Multimodal Large Language Models Multimodal Understanding Multimodal Generation+3

Ask PDF

Abstract

Multimodal Fusion

Explore similar work

Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning

Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis

Multimodal LLMs under Pairwise Modalities