Our aim is to advance the understanding of genuine audio-visual coordination in joint text-to-audio-video (T2AV) generation, with particular emphasis on semantic consistency. Evaluating audio-visual consistency in T2AV generation is nontrivial, as detailed text prompts obscure whether such consistency stems from genuine cross-modal coordination or merely from independent text following. To isolate the former, we introduce intrinsic jointness: the ability of audio and video to produce the same realization for details left underspecified by the prompt. Using semantically underspecified prompts (e.g., "musical instrument playing" rather than "electric guitar playing"), we investigate four recent models and reveal their limited intrinsic jointness. To address this limitation, we propose JointREPA, a simple representation alignment-based post-training method to facilitate cross-modal coordination. Since existing benchmarks use semantically detailed prompts, we introduce UNSAIDBench, a benchmark designed to evaluate intrinsic jointness under semantic underspecification. Experiments across two distinct models and three benchmarks, including UNSAIDBench, demonstrate that JointREPA consistently improves semantic consistency.
Figures & tables
Figure 1: Intrinsic jointness. Under detailed prompts, output consistency can arise from each modality independently following the text prompt alone, without audio–visual interaction. In contrast, underspecified prompts enable disentangling whether this consistency stems from mere prompt following or genuine audio–visual coordination. We define intrinsic jointness as the ability to maintain consistency under such underspecified prompts.
Figure 2: Q1. Semantic jointness under varying prompt specificity. We evaluate semantic consistency using three metrics (Agreement, AV-IB, and AVHScore) across different levels of prompt specificity. All metrics generally decrease as prompt specificity decreases (L3 → L2 → L1). At L1, Agreement scores for most models are close to the shuffled-pair baseline (marked by × ), obtained by randomly pairing audio and video generated with different seeds under the same prompt. This suggests that current T2AV models exhibit limited semantic jointness.
Figure 3: Q2. Semantic jointness in internal representations. We evaluate probe agreement across Transformer blocks (horizontal axis) at different generation steps and levels of prompt specificity. Probe agreement generally decreases as prompt specificity decreases (L3 → L2 → L1). At L1, probe agreement is close to the shuffled-pair baseline (dashed line). This suggests that limited semantic jointness is also reflected in the models’ internal representations.
L1
L2
L3
“A/An is vocalizing.”
animal
pet
dog / cat
farm animal
cow / sheep
wild animal
lion / wolf
“A person is playing a .”
musical instrument
string inst.
guitar / violin
Table 1: Simple diagnostic set for semantic jointness analysis. Prompts at L1, L2, and L3 share the same template but differ in the level of semantic specificity.
Figure 4: Q3. Analysis of the role of AV-attention in semantic coordination. (a) Semantic consistency with and without AV-attention. Blocking AV-attention causes substantial drops across all metrics in the LTX family, while inducing only minor changes in the JavisDiT family. (b) Semantic identity following under asymmetric prompting (L3 for one modality, L1 for the other). For both model families, the L1-conditioned modality mL1 follows the target identity specified zL3 substantially less reliably than under symmetric L3 prompting (both-L3). Together, these findings suggest that AV-attention is not effectively utilized for semantic coordination.
Figure 5: Overview of JointREPA. JointREPA aligns each modality’s features with pretrained encoder representations from both its own modality and the other modality.
AV-Consistency
AV-Synchrony
Text-Consistency
AV-Quality
Model
Size
AV-IB ↑
AVHScore ↑
JavisScore ↑
DeSync ↓
TV-IB ↑
TA-IB ↑
CLIP ↑
CLAP ↑
VQ ↑
AQ ↑
UNSAIDBench
JavisDiT++ [ 27 ]
2.1B
0.176
0.163
0.133
0.815
0.180
0.138
0.313
0.343
1.455
4.595
+ JointREPA
2.1B
0.207
0.196
0.164
0.804
0.174
0.163
0.310
0.369
1.655
4.628
LTX-2.3 [ 15 ]
22B
0.244
0.236
0.209
0.468
0.173
0.101
0.298
0.287
2.354
5.050
+ JointREPA
22B
0.302
0.288
0.261
0.440
0.169
0.124
0.298
0.327
2.256
4.852
Table 2: Main results on UNSAIDBench, JavisBench-mini, and VerseBench. We apply JointREPA to JavisDiT++ and LTX-2.3 and evaluate AV-Consistency, AV-Synchrony, Text-Consistency, and AV-Quality on three benchmarks. JointREPA consistently improves AV-Consistency across all benchmarks and backbone models.
Figure 6: Qualitative results on UNSAIDBench. Generated video frames and audio spectrograms from the base model (top) and its JointREPA variant (bottom). Given an underspecified shared factor per prompt, the base model realizes different semantic identities in audio and video, whereas JointREPA converges on the same identity.
Table 3: Ablations of fine-tuning losses on JavisDiT++. While the FM loss yields modest improvements and unimodal REPA provides larger gains, JointREPA achieves the best performance.
Figure 7: Revisiting Q1, Q2 and Q3 after applying JointREPA. (a) Q1. JointREPA improves all jointness metrics across specificity levels, peaking at L1—widening the Agreement margin over the shuffled baseline ( 0.11→0.17 ) and achieving larger average gains in AV-IB and AVHScore at lower specificity ( L3<L2<L1 ). (b) Q2. Intermediate representation probing reveals a pronounced increase in L1 Agreement around the middle layers where JointREPA is applied. (c) JointREPA increases the follow rate under asymmetric prompting (relative to Both-L3) from 26% to 47% and from 70% to 75%.
Joint audio-video generation models are rapidly approaching professional production quality, raising a central question: do they understand audio-visual physics, or merely generate plausible sounds and frames that violate real-world consistency? We introduce AV-Phys Bench, a benchmark for evaluating physical commonsense in joint audio-video generation. AV-Phys Bench tests models across three scene categories: Steady State, Event Transition, and Environment Transition. It covers physics-grounded subcategories drawn from real-world scenes, plus Anti-AV-Physics prompts that deliberately request physically inconsistent audio-video behavior. Each generation is evaluated along five dimensions: visual semantic adherence, audio semantic adherence, visual physical commonsense, audio physical commonsense, and cross-modal physical commonsense. Across three proprietary and four open-source models, we find that Seedance 2.0 performs best overall, but all models remain far from robust physical understanding. Performance drops sharply on event-driven and environment-driven transitions, and even strong proprietary systems collapse on Anti-AV-Physics prompts. We further introduce AV-Phys Agent, a ReAct-style evaluator that combines a multimodal language model with deterministic acoustic measurement tools, producing rankings that closely align with human ratings. Our results identify cross-modal physical consistency and transition-driven scene dynamics as key open challenges for joint audio-video generation.
Zijun Cui, Xiulong Liu, Hao Fang +8
University of Texas at Dallas · University of Washington · University of California, Los Angeles
Joint audio-video generation models have shown that unified generation yields stronger cross-modal coherence than cascaded approaches. However, existing models couple modalities throughout denoising via pervasive attention, treating high-level semantics and low-level details in a fully entangled manner. This is suboptimal for talking head synthesis: while audio and facial motion are semantically correlated, their low-level realizations (acoustic signals and visual textures) follow distinct rendering processes. Enforcing joint modeling across all levels causes unnecessary entanglement and reduces efficiency. We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone, while low-level refinement uses modality-specific decoders. A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space. Two lightweight diffusion transformer heads decode the hidden states into frame-level audio and video latents. Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality, achieving stronger cross-modal consistency than cascaded pipelines.
Zhen Ye, Xu Tan, Aoxiong Yin +8
Hong Kong University of Science and Technology · Independent Researcher · Zhejiang University +3
Current open-source diffusion models struggle to generate stable and synchronized audio-visual content, particularly in scenarios demanding complex semantic reasoning. The root cause is that existing methods rely on coarse text embeddings from off-the-shelf encoders to guide audio-video denoising, which discards fine-grained semantics and, critically, lacks a shared long-horizon plan, leading to uncoordinated denoising trajectories and fragile cross-modal alignment. We propose Baton, the first framework that introduces explicit semantic planning into joint video-audio generation. Our key insight is that complementing coarse text guidance with semantically rich, modality-aware planned tokens, jointly reasoned and mutually aligned before denoising, can simultaneously restore fine-grained semantic detail and establish a shared blueprint that coordinates both audio and video denoising trajectories. Concretely, Baton first introduces the VA-Planner, a multimodal language model equipped with dual semantic alignment towers, where learnable queries cross-attend to both video and audio features to produce a pair of semantically aligned video and audio planned tokens as keyframe-level blueprints. These planned tokens are injected into the diffusion backbone via cross-attention layers, providing temporally grounded guidance complementary to coarse text embeddings. Since planned tokens do not share one-to-one spatial-temporal correspondence with diffusion latents, we further propose Relative Semantic RoPE, a relative positional encoding that maps planned tokens and latents into a shared spatial-temporal coordinate frame, enabling each latent to accurately attend to its positionally corresponding semantic cues. Experiments on benchmarks show the effectiveness of Baton both qualitatively and quantitatively.