When Music Needs Meaning: Semantic Communication as a Transformative Enabler for the Musical Metaverse
Organizations: University of Applied Sciences and Arts of Southern Switzerland, Lugano, Switzerland · University of Trento, Trento, Italy · Centre Tecnologic de Telecomunicacions de Catalunya (CTTC/CERCA), Castelldefels, Spain. · New York University, New York, USA. ·
Abstract
The Musical Metaverse (MM) enables geographically distributed musicians, audiences, sound engineers, and AI agents to interact through synchronized audio, visual, kinematic, and haptic streams, imposing latency, synchronization, bandwidth, and scalability requirements that are difficult to satisfy with conventional bit-level communication. This paper argues that semantic communication, understood here as the transmission of musically meaningful information rather than raw audio-visual-haptic samples, provides a principled communication paradigm for the MM. We define the MM from a communication perspective and derive modality-specific require ments; propose a three-pillar framework for musical semantic communication comprising semantics extraction, semantic-aware orchestration, and cross-modal semantic synchronization; survey representative work across semantic communication, music AI and edge orchestration, identifying the gaps that prevent their integration; and propose a semantic-first reference architecture based on device-side semantic encoders, edge semantic routers, and cross-modal semantic synchronization. The resulting framework positions the MM as a concrete and demanding use case for 6G semantic communication.
Figures & tables
| Use case | Participants | Modalities | Latency focus | Semantic opportunity |
|---|---|---|---|---|
| UC1: Remote jamming | 3–5 musicians | Audio, kinematic | 30–40 ms audio; low, stable gesture delay | Protect rhythm-critical events under congestion; degrade secondary detail first. |
| UC2: Virtual concert | 10+ musicians; 1K–10K audience members | Audio, haptic, visual, kinematic | Per-modality targets of Table II , at audience scale | Transmit musical meaning once; receivers render to their capabilities (semantic multicast). |
| UC3: AI co-creation | One or more humans and AI agents | Audio, kinematic, semantic metadata | Role-dependent: in-loop vs. structural time scales | Exchange intent, structure, and expressive cues rather than waveforms. |
| Modality | Latency scope | Alignment landmark | Rate/fidelity variables |
|---|---|---|---|
| Audio | Source-to-ear monitoring or buffered audience playback. | Rendered onset relative to intended musical timing. | Channels, sample format, codec, and requested pitch/timbre detail. |
| Kinematic | Sensor capture to the receiving task or avatar action. | Defined gesture landmark; a gesture may precede sound. | Tracked joints/state variables, update rate, and quantization. |
| Haptic | Distinguish monitoring cues from closed-loop force interaction. | Measured cue onset/contact peak, including actuator delay. | Actuator channels, update rate, and cue versus force fidelity. |
| Visual | Distinguish interactive avatars from buffered concert video. | Displayed contact frame or other declared visual landmark. | Frame rate, resolution, scene complexity, and representation. |
| Pillar | Representative works | Coverage and main gap |
|---|---|---|
| P1: Semantics extraction | [ 9 , 10 , 21 , 22 , 23 , 11 , 24 , 25 , 26 ] | Codecs and features exist; assess named musical controls, causal delay, and receiver fidelity jointly. |
| P2: Semantic orchestration | [ 12 , 13 , 29 , 28 , 30 , 31 , 32 ] | Resource mechanisms exist; specify musical inputs and task-dependent policy evaluation. |
| P3: Semantic synchronization | [ 33 , 14 , 34 , 7 , 15 , 35 , 36 ] | Association and timing primitives exist; validate the combined event/state/rendering contract. |
| Requirement (Sec. II) | Integration question | Pillar | Proposed response to evaluate |
|---|---|---|---|
| Low-latency musical audio | Can a representation support the requested task within its full causal budget? | P1 | Measure extraction, communication, and rendering costs together. |
| Cross-modal synchronization | Which outputs belong to the same event, with what intended offsets? | P3 | Supply event identity and timing intent; measure actual output residuals. |
| Large multi-user scale | What is shared across recipients, beyond conventional replication? | P2 | Compare representation and adaptation costs under the same topology. |
| Bandwidth-constrained operation | Which information may be reduced for this receiver’s task? | P1 + P2 | Expose selectable detail and requests to an explicit priority policy. |
| Heterogeneous devices | Which local outputs preserve the requested function? | P1 + P2 | Declare renderer capability and evaluate preserved features, not plausibility alone. |
| Modality | : Feature semantics | : Structural semantics | : Intent and affect |
|---|---|---|---|
| Audio-semantic | Pitch contour, onset strength, beat times, instrument activity. | Chord progression, phrase boundaries, rhythmic hierarchy, key change. | Inferred tension, expressive timing intent, emotional valence, narrative arc. |
| Kinematic-haptic | Gesture velocity, finger pressure, bow position, impact force. | Gesture phrasing, repeated motion patterns, haptic event sequences. | Inferred expressive gesture purpose, anticipated action, perceived effort. |
| Visual-spatial | Avatar joints, facial action units, gaze direction. | Scene transitions, musician proximity, group formation, attention patterns. | Inferred collective emotion, audience engagement, musician–audience rapport. |
| Metric | Question answered | Key design choice |
|---|---|---|
| SMF | Are declared features preserved? | Specify decoded or rendered features; fixed losses/weights. |
| ESI | Are expected output relationships preserved? | Independent timing targets; missing pairs score zero. |
| ECS | Do inferred affect trajectories co-vary? | Common-sample correlation, not affective agreement. |
Semantic Communication
Semantic communication sends meaning- or task-relevant information so receivers can act without reconstructing every transmitted bit or signal.
See all 117 papers