Audio-video (AV) generation has recently made strong progress in perceptual quality and multimodal coherence, yet generating content with plausible motion-sound relations remains challenging. Existing methods often produce object motions that are visually unstable and sounds that are only loosely aligned with salient motion or contact events, largely because they lack an explicit motion-aware structure shared by video and audio generation. We present Tora3, a trajectory-guided AV generation framework that improves physical coherence by using object trajectories as a shared kinematic prior. Rather than treating trajectories as a video-only control signal, Tora3 uses them to jointly guide visual motion and acoustic events. Specifically, we design a trajectory-aligned motion representation for video, a kinematic-audio alignment module driven by trajectory-derived second-order kinematic states, and a hybrid flow matching scheme that preserves trajectory fidelity in trajectory-conditioned regions while maintaining local coherence elsewhere. We further curate PAV, a large-scale AV dataset emphasizing motion-relevant patterns with automatically extracted motion annotations. Extensive experiments show that Tora3 improves motion realism, motion-sound synchronization, and overall AV generation quality over strong open-source baselines. Project page: https://ali-videoai.github.io/tora3_page.
Figures & tables
Figure 1. Examples generated by Tora3. Tora3 generates audio-video content that follows prescribed trajectories with improved physical coherence, producing more plausible motion and better motion-sound alignment. Long trajectories are segmented for clarity. A gallery of example results generated by Tora3, showing video frames of objects moving along prescribed trajectories together with their aligned audio, illustrating physically coherent motion and motion-sound alignment.
Figure 2. Effect of trajectory guidance on audio-video synchronization. Tora3 better aligns footstep sounds with shoe-ground contact. The red arrow indicates the trajectory from 2.5 to 3.0 s ( 1◯ , 2◯ ) and from 3.5 to 4.0 s ( 3◯ ). Comparison of audio-video synchronization with and without trajectory guidance, where red arrows mark the walking trajectory across time intervals and show that footstep sounds are better aligned with shoe-ground contact when trajectory guidance is used.
Figure 3. The overall architecture of Tora3. Object trajectories provide a shared kinematic prior for both video and audio generation. The video branch uses trajectory-aligned motion injection, while the audio branch incorporates trajectory-derived kinematic features to better align sound with motion. Hybrid flow matching further improves trajectory-guided generation by balancing motion fidelity and local coherence. A block diagram of the Tora3 architecture showing a twin diffusion transformer with a video branch and an audio branch. Object trajectories feed a shared kinematic prior that supplies trajectory-aligned motion injection to the video branch and trajectory-derived kinematic features to the audio branch.
Video Quality
Audio Quality
Text Alignment
AV Synchronization
Motion Control
Method
# Params
AS ↑
FVD ↓
CE ↑
CU ↑
PC ↓
PQ ↑
CLAP ↑
CLIP-T ↑
FGAS ↑
ETE ↓
MAIC ↑
TrajError ↓
LTX-2 ( HaCohen et al., 2026 )
22.16B
4.31
989.6
3.28
6.17
2.43
6.73
0.31
0.29
0.187
0.284
0.41
-
Ovi ( Low et al., 2025 )
11.66B
4.40
887.7
3.30
6.01
1.85
6.44
0.43
0.30
0.156
0.301
0.37
-
MOVA ( Team et al., 2026 )
30.00B
4.63
849.8
3.05
6.31
1.77
6.95
0.46
0.31
0.201
0.236
0.49
-
AVControl ( Ben-Yosef et al., 2026 )
22.32B
4.52
829.6
3.29
6.22
2.18
6.79
0.39
0.30
0.209
0.214
0.55
19.95
Tora3
12.25B
4.61
784.1
3.34
6.43
1.81
7.09
0.44
0.31
0.234
0.181
0.63
12.13
Table 1. Comparison on video quality, audio quality, text alignment, motion control, and audio-video synchronization, including event-level and intensity-level motion-sound coherence metrics.
Figure 4. Qualitative comparison. Tora3 produces more stable visual motion and more coherent audio in both examples, including motion-consistent loudness variation for the moving motorcycle and better synchronized rolling and collision sounds for the rolling ball. Side-by-side qualitative comparison between Tora3 and baseline methods on a moving motorcycle example and a rolling ball example, with video frames and audio waveforms showing that Tora3 yields more stable motion and better synchronized loudness, rolling, and collision sounds.
Figure 5. Effect of motion speed on generated audio. Tora3 generates audio that varies consistently with motion speed. An illustration of a box being pulled at different speeds together with the corresponding generated audio waveforms, showing that faster motion produces louder sound with subtle timbral differences compared with slower motion.
Setting
AS ↑
FVD ↓
PQ ↑
FGAS ↑
ETE ↓
MAIC ↑
Neither
4.39
854.7
6.75
0.173
0.296
0.39
Video only
4.51
823.6
6.81
0.198
0.247
0.46
Audio only
4.42
845.2
6.89
0.209
0.221
0.61
Both (full)
4.47
811.8
6.93
0.225
0.193
0.66
Table 2. Effect of using trajectories as a shared kinematic prior for both modalities.
Method
AS ↑
FGAS ↑
ETE ↓
TrajError ↓
# Params
Tora-style ( Zhang et al., 2025c )
4.49
0.179
0.271
17.06
12.65B (+0.99B)
WanMove-style ( Chu et al., 2026 )
4.44
0.184
0.251
13.91
11.66B (+590K)
Ours
4.51
0.198
0.247
13.03
11.66B (+0)
Table 3. Ablation on motion representation designs for video-branch injection. Tora-style and WanMove-style denote variants inspired by Tora ( Zhang et al., 2025c ) and WanMove ( Chu et al., 2026 ) , respectively.
Kinematic Signal
PQ ↑
FGAS ↑
ETE ↓
MAIC ↑
None
6.75
0.173
0.296
0.39
r+v
6.79
0.191
0.254
0.48
r+v+a
6.86
0.202
0.229
0.56
r+v+a+∥v∥2+∥a∥2
6.89
0.209
0.221
0.61
Table 4. Effect of different kinematic state components on audio generation and motion-sound coherence.
Method
AS ↑
FVD ↓
TrajError ↓
FGAS ↑
w/o Hybrid flow matching
4.47
811.8
12.94
0.225
w/ Hybrid flow matching
4.61
784.1
12.13
0.234
Table 5. Effect of hybrid flow matching on video quality, trajectory following, and overall AV synchronization.
Joint audio-video generation aims to synthesize temporally synchronized and semantically coherent visual-acoustic content. However, existing open-source methods mainly rely on either dual-tower designs with posterior alignment or fully unified tri-modal designs that mix textual context, audio and video in one shared space. The former weakens fine-grained audio-video co-evolution, while the latter couples semantic conditioning with low-level synchronization. To address these limitations, we propose NAVA, a Native Audio-Visual Alignment framework for joint audio-video generation. NAVA is built upon context-conditioned native audio-visual alignment: it first establishes audio-video correspondence in a dedicated interaction space, and then uses external context to condition the joint denoising process. Specifically, NAVA is instantiated with an Align-then-Fuse MMDiT architecture, which transitions from modality-aware audio-video alignment to modality-shared joint denoising. Furthermore, we introduce Timbre-in-Context Conditioning to associate reference timbre cues with corresponding speech spans to achieve controllable speech timbre. Experiments on Verse-Bench and Seed-TTS, together with a user study, demonstrate that NAVA achieves superior video quality, precise audio-visual synchronization, competitive audio quality, and stronger reference-timbre controllability using only 6.3B parameters.
Joint audio-video generation models have shown that unified generation yields stronger cross-modal coherence than cascaded approaches. However, existing models couple modalities throughout denoising via pervasive attention, treating high-level semantics and low-level details in a fully entangled manner. This is suboptimal for talking head synthesis: while audio and facial motion are semantically correlated, their low-level realizations (acoustic signals and visual textures) follow distinct rendering processes. Enforcing joint modeling across all levels causes unnecessary entanglement and reduces efficiency. We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone, while low-level refinement uses modality-specific decoders. A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space. Two lightweight diffusion transformer heads decode the hidden states into frame-level audio and video latents. Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality, achieving stronger cross-modal consistency than cascaded pipelines.
Zhen Ye, Xu Tan, Aoxiong Yin +8
Hong Kong University of Science and Technology · Independent Researcher · Zhejiang University +3
Text-to-audio-video (T2AV) generation is central to applications such as filmmaking and world modeling. However, current models often fail to produce physically plausible sounds. Previous benchmarks primarily focus on audio-video temporal synchronization, while largely overlooking explicit evaluation of audio-physics grounding, thereby limiting the study of physically plausible audio-visual generation. To address this issue, we present PhyAVBench, the first benchmark that systematically evaluates the audio-physics grounding capabilities of T2AV, image-to-audio-video (I2AV), and video-to-audio (V2A) models. PhyAVBench offers PhyAV-Sound-11K, a new dataset of 25.5 hours of 11,605 audible videos collected from 184 participants to ensure diversity and avoid data leakage. It contains 337 paired-prompt groups with controlled physical variations that drive sound differences, each grounded with an average of 17 videos and spanning 6 audio-physics dimensions and 41 fine-grained test points. Each prompt pair is annotated with the physical factors underlying their acoustic differences. Importantly, PhyAVBench leverages paired text prompts to evaluate this capability. We term this evaluation paradigm the Audio-Physics Sensitivity Test (APST) and introduce a novel metric, the Contrastive Physical Response Score (CPRS), which quantifies the acoustic consistency between generated videos and their real-world counterparts. We conduct a comprehensive evaluation of 17 state-of-the-art models. Our results reveal that even leading commercial models struggle with fundamental audio-physical phenomena, exposing a critical gap beyond audio-visual synchronization and pointing to future research directions. We hope PhyAVBench will serve as a foundation for advancing physically grounded audio-visual generation. Prompts, ground-truth, and generated video samples are available at https://github.com/imxtx/PhyAVBench.