Preserving multimodal signals across agent boundaries is necessary for accurate cross-modal reasoning, but it is not sufficient. We show that modality-native routing in Agent-to-Agent (A2A) networks improves task accuracy by 20 percentage points over text-bottleneck baselines, but only when the downstream reasoning agent can exploit the richer context that native routing preserves. An ablation replacing LLM-backed reasoning with keyword matching eliminates the accuracy gap entirely (36% vs. 36%), establishing a two-layer requirement: protocol-level routing must be paired with capable agent-level reasoning for the benefit to materialize. We present MMA2A, an architecture layer atop A2A that inspects Agent Card capability declarations to route voice, image, and text parts in their native modality. On CrossModal-CS, a controlled 50-task benchmark with the same LLM backend, same tasks, and only the routing path varying, MMA2A achieves 52% task completion accuracy versus 32% for the text-bottleneck baseline (95% bootstrap CI on ΔTCA: [8, 32] pp; McNemar's exact p=0.006). Gains concentrate on vision-dependent tasks: product defect reports improve by +38.5 pp and visual troubleshooting by +16.7 pp. This accuracy gain comes at a 1.8× latency cost from native multimodal processing. These results suggest that routing is a first-order design variable in multi-agent systems, as it determines the information available for downstream reasoning.
Figures & tables
Figure 1: MMA2A deployment architecture. The Modality-Aware Router inspects Agent Card capabilities and forwards parts in their native modality. Annotations show the A2A Part type used for each route.
Figure 2: Information flow comparison. Text-BN (top) transcodes all non-text parts, losing prosodic and spatial signals. MMA2A (bottom) preserves native modality, delivering richer evidence to the shared LLM reasoning step.
Category
n
Voice
Image
Text
Product defect report
13
✓
✓
✓
Assembly guidance
12
✓
✓
Visual troubleshooting
12
✓
✓
Warranty claim
13
✓
✓
✓
Table 1: CrossModal-CS task categories and modality requirements.
System
TCA (%)
Latency (s)
BW (KB/task)
Native routing (%)
Text-BN
32.0
7.19 ± 4.46
329
50.5
MMA2A (Ours)
52.0
13.04 ± 6.39
330
81.7
Δ
+20.0 pp
+5.85 s
+0.5%
+31.2 pp
Table 2: Main results on CrossModal-CS (50 paired tasks). MMA2A achieves substantially higher accuracy at the cost of increased latency from native multimodal processing.
Decision Method
Text-BN TCA
MMA2A TCA
Δ
Keyword matching (heuristic)
36%
36%
0 pp
LLM reasoning (Gemini)
32%
52%
+20 pp
Table 3: Ablation: keyword matching vs. LLM reasoning for the final decision step. With keyword matching, the routing strategy has no effect on accuracy.
Figure 3: Causal diagram of the two-layer requirement. Routing strategy determines input fidelity, which enables accuracy gains—but only when reasoning capability (moderator) can exploit the richer evidence. The ablation confirms that blocking the moderator eliminates the effect.
Category
n
Text-BN
MMA2A
Δ
Product defect report
13
7.7
46.2
+38.5
Visual troubleshooting
12
75.0
91.7
+16.7
Assembly guidance
12
41.7
58.3
+16.6
Warranty claim
13
7.7
15.4
+7.7
Table 4: TCA (%) by task category. The largest gains appear in vision-dependent categories where text transcoding loses the most signal.
Modality
MMA2A
Text-BN
Native
Transcode
Native
Transcode
Voice
40 (100%)
0 (0%)
0 (0%)
40 (100%)
Image
28 (70%)
12 (30%)
0 (0%)
40 (100%)
Text
110 (80%)
28 (20%)
110 (80%)
28 (20%)
Total
178 (82%)
40 (18%)
110 (50%)
108 (50%)
Table 5: Routing decisions by modality. MMA2A achieves 100% native routing for voice and 70% for images; Text-BN forces all non-text parts through transcoding.
Category
Mean (s)
Median (s)
Text-BN
MMA2A
Text-BN
MMA2A
Assembly guidance
9.67
9.66
8.47
8.81
Product defect report
3.96
16.55
3.74
18.19
Visual troubleshooting
11.03
14.51
9.50
13.61
Warranty claim
4.61
11.31
4.66
8.77
Overall
7.19
13.04
5.93
9.42
Table 6: Latency statistics (seconds) by category. MMA2A is consistently slower due to native multimodal processing, with the cost proportional to the complexity of the multimodal input.
Failure Mode
n
Layer
Description
Policy lookup failure
11
Reasoning
Correct evidence, wrong policy match
Action granularity confusion
6
Reasoning
Semantically adjacent action selected
Overconfident visual grounding
4
Routing × Reasoning
Richer input triggers over-action
Insufficient context
3
Routing
Too little evidence reaches the agent
Table 7: MMA2A failure modes, classified by manual inspection of predicted-vs-expected action pairs and agent response traces. The Layer column identifies which architectural layer is responsible for the failure.
Figure 4: Information topologies induced by routing strategy. (a) Text-BN funnels all modalities through a lossy text bottleneck, imposing a uniform but degraded evidence surface on all agents. (b) MMA2A routes each modality natively to a capable agent, creating a heterogeneous topology with higher-fidelity inputs. Error modes from Table 7 are mapped to the architectural layer responsible: the vast majority of remaining failures occur downstream of routing, in the reasoning layer.
Robust embodied navigation relies on complementary sensory cues. However, high-quality and well-aligned multi-modal data is often difficult to obtain in practice. Training a monolithic model is also challenging as rich multi-modal inputs induce complex representations and substantially enlarge the policy space. Cross-modal collaboration among lightweight modality-specialized agents offers a scalable paradigm. It enables flexible deployment and parallel execution, while preserving the strength of each modality. In this paper, we propose CRONA, a Multi-Agent Reinforcement Learning (MARL) framework for Cross-Modal Navigation. CRONA improves collaboration by leveraging control-relevant auxiliary beliefs and a centralized multi-modal critic with global state. Experiments on visual-acoustic navigation tasks show that multi-agent methods significantly improve performance and efficiency over single-agent baselines. We find that homogeneous collaboration with limited modalities is sufficient for short-range navigation under salient cues; heterogeneous collaboration among agents with complementary modalities is generally efficient and effective; and navigation in large, complex environments requires both richer multi-modal perception and increased model capacity.
Shuo Liu, Xinzichen Li, Christopher Amato
Khoury College of Computer Sciences Northeastern University Boston, MA, 02120
Current evaluation frameworks and benchmarks for LLM powered agents focus on text chat driven agents, these frameworks do not expose the persona of user to the agent, thus operating in a user agnostic environment. Importantly, in customer experience management domain, the agent's behaviour evolves as the agent learns about user personality. With proliferation of real time TTS and multi-modal language models, LLM based agents are gradually going to become multi-modal. Towards this, we propose the MM-tau-p2 benchmark with metrics for evaluating the robustness of multi-modal agents in dual control setting with and without persona adaption of user, while also taking user inputs in the planning process to resolve a user query. In particular, our work shows that even with state of-the-art frontier LLMs like GPT-5, GPT 4.1, there are additional considerations measured using metrics viz. multi-modal robustness, turn overhead while introducing multi-modality into LLM based agents. Overall, MM-tau-p2 builds on our prior work FOCAL and provides a holistic way of evaluating multi-modal agents in an automated way by introducing 12 novel metrics. We also provide estimates of these metrics on the telecom and retail domains by using the LLM-as-judge approach using carefully crafted prompts with well defined rubrics for evaluating each conversation.
Current multimodal fusion approaches, particularly those based on static Mixture-of-Experts (MoE) architectures, often struggle to provide the adaptive and efficient collaborative reasoning required by complex real-world applications. We introduce the Dynamic Agent-based Interaction Network (DAIN), which reconceptualizes multimodal fusion as a dynamic, multi-agent collaborative process. DAIN employs a context-aware Meta-Controller that dynamically schedules sparse activation of specialized interaction agents and orchestrates compressed inter-agent communication for consensus-building. The framework is guided by a multi-objective loss function that jointly optimizes task accuracy, agent specialization, and operational efficiency through sparse activation and communication regularization. Comprehensive evaluations across five diverse benchmarks -- ADNI, MIMIC-IV, MM-IMDB, CMU-MOSI, and ENRICO -- establish DAIN as a new state-of-the-art, delivering significant performance improvements including a 2.6% accuracy gain on ADNI. Ablation studies verify the critical roles of both dynamic scheduling and agent communication. Furthermore, DAIN offers enhanced interpretability by exposing context-dependent agent roles and collaboration patterns while maintaining computational efficiency through sample-wise sparse agent activation. Our work demonstrates the promise of dynamic, agent-based paradigms for multimodal reasoning.