cs.MASep 16, 2026

BirdsongChat: A Hybrid Multi-Agent Framework for Multimodal Embodied Behavior Simulation

Authors: Callie C. Liao, Duoduo Liao, Ellie L. Zhang

Organizations: IntelliSky McLean, VA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal embodied systems require translating human intentions into interpretable and coordinated behaviors across heterogeneous modalities. However, existing multimodal agents often rely on implicit representations, limiting controllability and cross-modal consistency. We present a hybrid multi-agent framework for interactive multimodal behavior simulation that bridges semantic reasoning and physical execution through a Unified Parameter Representation (UPR). LLM-based reasoning agents transform multimodal inputs into UPR, which encodes behavioral states and interpretable control parameters for simulation agents generating synchronized 3D motion, spatialized soundscapes, and environmental behaviors. We develop BirdsongChat as a prototype implementation of the proposed framework, using interactive avian behavior simulation as a testbed that tightly couples motion, vocalization, and environmental context. BirdsongChat is evaluated on text- and image-guided scenarios involving species, behaviors, affective states, environments, and multi-bird interactions. The system achieves normalized scores of 94.4% for cross-modal coherence, 100% for affective consistency, and 92.6% for generation consistency. These results demonstrate that an explicit intermediate representation effectively bridges semantic reasoning and physical execution, improving controllability and multimodal synchronization. The proposed framework thus offers a generalizable design principle for embodied AI systems requiring interpretable semantic-to-physical coordination across modalities, with potential applications in bio-inspired ecoacoustics, swarm robotics, virtual environments, and creative multimedia.

Figures & tables

Fig. 1 : BirdsongChat: Interactive Multimodal Control and Generation of Synchronized 3D Bird Motion and Soundscape . Foundation models perform semantic interpretation of species, behavior, and context from text and images. The resulting UPR parameters control algorithmic simulators for motion, sound, and environment generation.
Fig. 2 : Overview of the BirdsongChat framework for physically grounded multimodal generation and control . The architecture bridges high-level semantic reasoning with low-level physical execution through a hierarchical design including four functional layers, maintaining interpretability, physical plausibility, and cross-modal coherence.
ParameterDescriptionBehavioral Effect
Kinematic
x⃗0\vec{x}_{0}Equilibrium positionNominal CoM location
Δx⃗\Delta\vec{x}Oscillation amplitudesPath curvature
λ\lambdaRelaxation ratesAgility
σ\sigmaNoise amplitudesStochastic variability
v0v_{0}Preferred horizontal speedTarget forward velocity
TABLE I : Kinematic and Acoustic Control Parameters
DimensionValuesDescription
Bird Speciesowl, nightingale, lark, warbler, sparrow, finch, ravenDistinct vocal patterns for species-specific acoustic assessment
Environmentsnight, moon, moonlight, daytime, sun, forest, storm, rain, wind, sand, homeGuides scene composition and 3D auditory spatialization
Emotional Stateshappy, fear, sadModulates motion (speed, trajectory) and sound (pitch, rhythm, intensity)
TABLE II : Category Definitions
ScoreCross-Modal CoherenceAffective ConsistencyGeneration Consistency
3Strong motion, sound, and environment alignment; physically plausibleBehavior and sound match the specified affective descriptorStable outputs across runs with natural variation
2Minor mismatch but overall coherentPartial affective correspondenceNoticeable variation with preserved semantics
1Major mismatch or implausible behaviorWeak or unclear affective correspondenceLarge variation affecting consistency
0Failed multimodal generationNo affective correspondenceInconsistent outputs across runs
TABLE III : Score definitions for cross-modal coherence, affective consistency, and generation consistency (0–3 scale).
IDText PromptEnvironmentAffectSpeciesCorrectCoh.Aff.Gen.
1The moonlight shines brightly on the damp forest floor.Night, Moon, ForestFear/SadNightingale, OwlYes333
2The backyard forest sings harmonically with the newly bloomed flowers.Daytime, Sun, ForestHappySparrow, Finch, RavenYes333
3The birds sing happily with another under the gleaming sun.Sun, Wind, ForestHappySparrow, Finch, RavenYes333
4In a gloomy dark night, birds chirp silently in the forbidden forest of fear.Night, ForestFearNightingale, OwlYes333
5The birds shudder as a haunting call echoes from the distance.Night, MoonFearOwl, NightingaleYes333
6Everyone in the forest chirps excitedly as the new dawn comes.Sun, Wind, ForestHappySparrow, Raven, FinchYes333
TABLE IV : Text prompt alignment effectiveness with per-sample scores (0–3 scale).
IDImage ContextEnvironmentAffectSpeciesCoh.Aff.Gen.
1SunsetSun, Forest, StormHappyRaven, Sparrow, Finch232
2Beach with sand and seaSun, Sand, DaytimeHappyLark333
3Beach with sand, sea, and pine treeSun, SandHappyLark333
4Nighttime moon and seaNight, Moon, SeaFearNightingale, Owl333
5ThunderstormStorm, WindFearRaven, Sparrow333
6Foggy golf courseFog, ForestHappyNightingale, Owl132
TABLE V : Image prompt evaluation results with per-sample scores (0–3 scale).
Fig. 3 : Experimental image prompt examples.
Fig. 4 : BirdsongChat interface and examples. Users can directly copy prompts to try the AWS web-based prototype system.
MetricAverageMinMaxScore (%)
Text Prompts (12 prompts, 60 runs)
Cross-modal Coherence3.0033100%
Affective Consistency3.0033100%
Generation Consistency2.832394.4%
Overall2.942398.1%
Image Prompts (6 prompts, 30 runs)
TABLE VI : Quantitative evaluation results (0–3 scale). Each prompt is executed five times to evaluate semantic alignment and generation consistency.
More in this topic

Multimodal Agents

Multimodal agents perceive and reason across text, images, audio, and video to use tools, interact with interfaces, and act in the world.

See all 389 papers

Explore similar work

CardsList
Mar 24, 2026

A Multimodal Framework for Human-Multi-Agent Interaction

Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.

Shaid Hasan, Breenice Lee, Sujan Sarker +1

University of Virginia

Human-Robot InteractionHuman-AI InteractionMultimodal AgentsMulti-Agent Coordination
AskPDF
Feb 27, 2026

U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation

Full-stack multimodal interaction in real-time is a central goal in building intelligent embodied agents capable of natural, dynamic communication. However, existing systems are either limited to unimodal generation or suffer from degraded reasoning and poor cross-modal alignment, preventing coherent and perceptually grounded interactions. In this work, we introduce U-Mind, the first unified system for high-intelligence multimodal dialogue that supports real-time generation and jointly models language, speech, motion, and video synthesis within a single interactive loop. At its core, U-Mind implements a Unified Alignment and Reasoning Framework that addresses two key challenges: enhancing cross-modal synchronization via a segment-wise alignment strategy, and preserving reasoning abilities through Rehearsal-Driven Learning. During inference, U-Mind adopts a text-first decoding pipeline that performs internal chain-of-thought planning followed by temporally synchronized generation across modalities. To close the loop, we implement a real-time video rendering framework conditioned on pose and speech, enabling expressive and synchronized visual feedback. Extensive experiments demonstrate that U-Mind achieves state-of-the-art performance on a range of multimodal interaction tasks, including question answering, instruction following, and motion generation, paving the way toward intelligent, immersive conversational agents.

Xiang Deng, Feng Gao, Yong Zhang +5

Tsinghua University · Meituan

Multimodal AgentsUnified Multimodal ModelsReal-Time Interactive Video GenerationMultimodal Reasoning
AskPDF
Jun 30, 2026

MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments

Recent multimodal large language models (MLLMs) have strong potential as embodied agents, but their ability to collaborate in visually grounded environments remains underexplored. To address this gap, we introduce MECoBench, a multimodal embodied cooperation benchmark with an evaluation platform spanning diverse real-world tasks, two cooperation structures, and three collaboration modes. Through extensive experiments across various MLLMs, we summarize three key findings: (i) Collaboration generally improves embodied task completion, but its benefits depend on balancing collaborative gains against coordination complexity. (ii) Communication is essential to collaboration gains, while the best collaboration mode depends on team size and model capability. (iii) Moreover, collaboration improves robustness under noisy priors and exploration conditions. Generally, MECoBench provides a systematic testbed for understanding the mechanisms and limits of multimodal embodied collaboration. Code and dataset are available at https://github.com/q-i-n-g/MECoBench.

Qingyun Liu, Jiwen Zhang, Jingyi Hu +2

1Fudan University · 3The Chinese University of Hong Kong · 2Shanghai Innovation Institute

Multi-Agent CollaborationAI Agent BenchmarksMultimodal AgentsMultimodal Model Evaluation
AskPDF