Text-to-video generation is expensive, so only a few samples are typically produced per prompt. In this low-sample regime, maximizing the value of each batch requires high cross-video diversity. Recent methods improve diversity for image generation, but for videos they often degrade within-video temporal consistency and require costly backpropagation through a video decoder. We propose a joint-sampling framework for flow-matching video generators that improves batch diversity while preserving temporal consistency. Our approach applies diversity-driven updates and then removes only the components that would decrease a temporal-consistency objective. To avoid image-space gradients, we compute both objectives with lightweight latent-space models, avoiding video decoding and decoder backpropagation. Experiments on a state-of-the-art text-to-video flow-matching model show diversity close to strong joint-sampling baselines while substantially improving temporal consistency and color naturalness. Our code is available at https://github.com/XinshuangL/Diverse-Video.
Figures & tables
Fig. 1 : Joint video generation with enhanced cross-video diversity and preserved intra-video temporal consistency based on latent-space embedding and interpolation.
Fig. 2 : Illustration of training procedure for latent-space embedding and interpolation models.
Method
Vendi-v ↑
Vendi-f ↑
MSE ↓
CNI ↑
IID
1.55(1)
1.69(1)
0.0010(1)
0.67(1)
DPP
1.61(1)
1.81(1)
0.0028(2)
0.65(2)
Particle Guidance
1.61(1)
1.82(1)
0.0028(2)
0.65(2)
DiverseFlow
1.61(1)
1.81(2)
0.0029(2)
0.65(2)
Ours
1.60(1)
1.76(1)
0.0019(1)
0.68(1)
TABLE I : Video generation results. Metrics are reported as mean(uncertainty) with 95% confidence intervals; uncertainty aligns with the last digit of the mean.
Fig. 3 : Model behavior across flow-matching steps: (a) Extrapolation of x^1=xt+(1−t)vθ(xt,t) from intermediate latent states and decoding to video frames. (b)–(e) Embedding alignment losses for our latent embedding models, compared to using a latent-mean baseline. (f) Frame interpolation loss for Mc , compared to simple baselines (previous frame, next frame, and mean of both).
Diversity-v
ConsisReg
Vendi-v ↑
Vendi-f ↑
MSE ↓
CNI ↑
✗
✗
1.61(1)
1.77(1)
0.0017(1)
0.68(1)
✗
✓
1.60(1)
1.76(1)
0.0016(1)
0.68(1)
✓
✗
1.61(1)
1.77(1)
0.0021(1)
0.68(1)
✓
✓
1.60(1)
1.76(1)
0.0019(1)
0.68(1)
TABLE II : Ablation results. Metrics are reported as mean(uncertainty) with 95% confidence intervals.
Generating geometrically consistent videos remains an open challenge: text-to-video diffusion models trained on web-scale data treat geometry only implicitly, leading to object deformation, texture drift, and non-rigid backgrounds under camera motion. Existing solutions either improve consistency as a byproduct, apply only to static scenes or realign the latent space of the model completely. We introduce a geometry-consistency reward that directly measures whether motion in a generated video is compatible with a coherent scene. Our key insight is that in physically consistent videos, background motion should be explainable by rigid camera-induced flow, while independently moving objects should preserve appearance identity along motion trajectories. We operationalize this using optical flow, depth--pose predictions, and feature-based correspondence to separate rigid and dynamic regions and evaluate their respective consistency. Integrating this reward with reinforcement fine-tuning transforms geometric consistency from an emergent property into an explicit optimization objective for video generators. The approach is model agnostic and applies to diverse dynamic scenes containing both camera and object motion. Experiments show substantial reductions in temporal geometric artifacts over strong baselines while preserving perceptual quality. Code and model weights are published.
Flow Matching has enabled robust text-to-video generation via latent ODE sampling. However, velocity approximation and numerical discretization errors inevitably accumulate, causing sampling trajectories to drift. Consequently, generated videos often suffer from severe spatiotemporal inconsistencies. Nevertheless, directly correcting these drifted, noisy latents is challenging: (i) timestep-dependent noise obscures reliable structural cues; (ii) spatial interventions risk disrupting intricate local geometry while incurring heavy computational costs. To address this, we propose Spectral Lookahead Rectification (SpecLoR), a plug-and-play inference method that bypasses noise via lookahead prediction, and circumvents spatiotemporal entanglement by shifting corrections to the frequency domain, where universal statistical priors of natural videos are readily available. First, during early sampling stages, SpecLoR looks ahead to estimate the clean latent zt,0 and computes its 3D spatiotemporal spectrum. Next, SpecLoR rectifies the amplitude spectrum to match the prior, leaving the phase intact. Finally, the corrected state is re-noised to resume ODE integration. Experiments on Wan2.2 demonstrate that SpecLoR significantly reduces physical artifacts and enhances motion coherence across multiple benchmarks with minimal computational overhead (4 additional NFEs).
Xu Zhang, Yu Lu, Ruijie Quan +3
ReLER, College of Artificial Intelligence, Zhejiang University · Huawei Central Research Institute
Streaming video generation holds strong potential for world modeling, where future frames must be inferred online sequentially to form a continuous video stream. However, streaming video diffusion models introduce a fundamental train-inference mismatch: inference follows a specialized denoising order, whereas advanced training strategies typically require diverse noise-level configurations. To address this trade-off between train-inference consistency and training coverage, we reformulate the video diffusion sampling as a frame-indexed stochastic process over noise levels. Within this stochastic process space, we construct a continuous training trajectory along which the sampling schedule progressively evolves from independent sampling to inference-consistent sampling. We further introduce a joint calibration algorithm and a temporal correlative sampling algorithm to ensure trajectory smoothness and cross-frame correlation. Building on these designs, we propose Stream Forcing, a unified training framework for streaming video generation that balances training sufficiency and inference efficiency. Extensive experiments demonstrate that Stream Forcing significantly improves generation quality with a 36.6% FVD improvement on the UCF-101 benchmark. Furthermore, our method facilitates robust zero-shot extrapolation to long-horizon video generation with a 27.9% FVD improvement on the UCF-101 benchmark.
Yueting Zhu, Yuehao Song, Kaicheng Zhang +5
Huazhong University of Science & Technology · Anyverse Dynamics · Horizon Robotics