JVT: Just Video Transformer makes pixel-space video diffusion simple
Organizations: Zhejiang University · Shanghai Innovation Institute · Shanghai AI Laboratory
Abstract
Direct diffusion modeling of high-dimensional raw video data is challenging, and existing pixel-space methods often rely on specialized architectures. We introduce JVT (Just Video Transformer), a simple large-patch DiT-style Transformer for pixel-space video diffusion, without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net-style skip connections. We systematically examine video patch embeddings, register tokens, and adaptive layer normalization to identify design choices that balance generation quality and computational efficiency. The resulting JVT design achieves generation quality competitive with latent diffusion baselines in both class-conditioned and unconditional settings. The pixel-space formulation of JVT also naturally supports joint generation of pixel-aligned modalities. We demonstrate this capability through camera-conditioned RGB-disparity generation, which improves camera following relative to RGB-only baselines and yields explicit scene geometry without an additional reconstruction model at inference. These results highlight the potential of simple DiT-style Transformers for pixel-space video diffusion, and we hope our findings offer useful insights and practical guidance for future research.
Figures & tables
| Method | Res. | Cond. | Uncond. | |||
|---|---|---|---|---|---|---|
| UCF-101 | UCF-101 | FaceForensics | SkyTimelapse | Taichi-HD | ||
| MoCoGAN ( Tulyakov et al., 2018 ) | – | 2886.9 | 124.7 | 206.6 | – | |
| MoCoGAN-HD ( Tian et al., 2021 ) | – | 1729.6 | 111.8 | 164.1 | 144.7 | |
| VideoGPT ( Yan et al., 2021 ) | – | 2880.6 | 185.9 | 222.7 | – | |
| StyleGAN-V ( Skorokhodov et al., 2022 ) | – | 1431.0 | 47.41 | 79.52 | – | |
| MoStGAN-V ( Shen et al., 2023 ) | – | 1380.3 | 39.7 | 65.3 | – | |
| Embedding | Tubelet | Patch-first | Frame-first | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| - | 32 | 64 | 128 | 16 | 32 | 64 | 8 | 16 | 32 | |
| FVD ( ) | 145.0 | 147.4 | 135.6 | 142.3 | 143.6 | 135.0 | 143.5 | 151.2 | 149.7 | 146.6 |
| Final loss ( ) | 0.0151 | 0.0150 | 0.0147 | 0.0143 | 0.0146 | 0.0143 | 0.0141 | 0.0146 | 0.0146 | 0.0142 |
| Denoiser Params (M) | 682.2 | 679.2 | 679.7 | 680.8 | 678.7 | 678.8 | 679.0 | 681.0 | 683.3 | 688.1 |
| Denoiser FLOPs (TFLOPs) | 1.089 | 1.083 | 1.084 | 1.086 | 1.082 | 1.082 | 1.082 | 1.086 | 1.091 | 1.101 |
| Denoiser Latency (ms, ) | 16.07 | 15.96 | 15.99 | 15.58 | 15.53 | 15.59 | 15.59 | 15.42 | 15.64 | 15.89 |
| Guidance | Sampling | FVD@2048 ( ) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| CFG | RG | Solver | Steps | NFE | No reg. | ||||
| ✓ | ✗ | Euler | 50 | 100 | 136.5 | 137.9 | 138.6 | 137.2 | 140.4 |
| ✓ | ✗ | Euler | 100 | 200 | 145.2 | 135.1 | 136.8 | 138.0 | 142.5 |
| ✓ | ✗ | Heun | 50 | 200 | 167.4 | 158.7 | 149.1 | 157.2 | 160.8 |
| ✓ | ✓ | Euler | 50 | 150 | – | 129.2 | 130.4 | 133.3 | 132.0 |
| Variant | AdaLN condition | FVD ( ) | |
|---|---|---|---|
| baseline | – | 136.5 | |
| w/ | 138.6 | ||
| w/o noise level | – | 140.6 | |
| w/o AdaLN | – | 156.3 |
| Method | Objective | FVD ( ) | RRA ( ) | RTA ( ) | AUC ( ) | |||
|---|---|---|---|---|---|---|---|---|
| @5 | @30 | @5 | @30 | @5 | @30 | |||
| DFoT ( Song et al., 2025 ) | RGB | 35.3 | 99.9 | 100 | 77.7 | 97.4 | 58.3 | 87.2 |
| Geometry Forcing ( Wu et al., 2026 ) | RGB | 46.8 | 99.8 | 100 | 73.6 | 96.4 | 52.1 | 84.7 |
| LIVE ( Huang et al., 2026b ) | RGB | 42.5 | 99.1 | 100 | 49.8 | 92.6 | 28.4 | 73.3 |
| JVT (Ours) | RGB | 27.0 | 99.9 | 100 | 72.4 | 96.0 | 52.8 | 84.1 |
| JVT (Ours) | RGB-disparity | 27.5 | 99.9 | 100 | 79.5 | 98.4 | 58.6 | 88.6 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Standard RGB | Camera RGB | Camera RGB–disp. | |
| Backbone | JVT-XL | JVT-L | JVT-L, joint I/O |
| Depth / width / heads | 28 / 1,152 / 16 | 24 / 1,024 / 16 | 24 / 1,024 / 16 |
| Video size | |||
| Input/output channels | 3 | 3 | 4 |
| Spatial patch size | |||
| Temporal factor | 4 | 1 | 1 |
| Condition | Class-conditioned | Unconditional | Unconditional | Camera control |
|---|---|---|---|---|
| Dataset | UCF-101 | UCF-101 | FFS / Sky / Taichi | RealEstate10K |
| Guidance | CFG or CFG+RG | RG | None | HG |
| Guidance scale | CFG 4, RG 0.6 | 0.8 | - | 4 |
| Solver | Euler | Heun | Heun | Euler |
| Steps | 50 | |||
| Noise shift | 4 | |||
| Embedding | In-context | Tokens | Params (M) | FLOPs | Peak (GiB) | |||
|---|---|---|---|---|---|---|---|---|
| Total (TFLOPs) | Embed. (GFLOPs) | Embed. share (%) | ||||||
| Tubelet | – | yes | 1056 | 682.2 | 1.089 | 7.248 | 0.666 | 2.674 |
| Tubelet | 32 | yes | 1056 | 679.2 | 1.083 | 1.107 | 0.102 | 2.650 |
| Tubelet | 64 | yes | 1056 | 679.7 | 1.084 | 2.215 | 0.204 | 2.652 |
| Tubelet | 128 | yes | 1056 | 680.8 | 1.086 | 4.429 | 0.408 | 2.659 |
| Patch-first | 16 | yes | 1056 | 678.7 | 1.082 | 0.252 | 0.023 | 2.648 |
| Embedding | In-context | Guidance | Fwd. | Latency (ms) | Peak (GiB) | Est. FLOPs (TFLOPs) |
|---|---|---|---|---|---|---|
| Tubelet (plain) | yes | None | 1 | 2.689 | 54.4 | |
| Tubelet (plain) | yes | CFG | 2 | 2.751 | 108.9 | |
| Tubelet (plain) | yes | RG | 2 | 2.707 | 108.9 | |
| Tubelet (plain) | yes | CFG+RG | 3 | 2.768 | 163.3 | |
| Tubelet (plain) | no | None | 1 | 2.689 | 53.2 | |
| Tubelet (plain) | no | CFG | 2 | 2.751 | 106.3 |
| RG | NFE | Solver | Steps | UCF-101 | FaceForensics | SkyTimelapse | Taichi-HD | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No reg. | No reg. | No reg. | No reg. | ||||||||||||
| ✗ | 100 | Heun | 50 | 413.5 | 378.6 | 474.2 | 32.9 | 37.7 | 33.1 | 46.6 | 54.9 | 52.6 | 119.1 | 144.6 | 127.1 |
| ✗ | 200 | Euler | 200 | 421.2 | 392.4 | 484.5 | 35.2 | 40.2 | 34.9 | 49.0 | 58.3 | 55.3 | 114.5 | 143.6 | 124.3 |
| ✗ | 200 | Heun | 100 | 387.7 | 354.9 | 451.5 | 34.0 | 39.0 | 34.0 | 45.5 | 54.0 | 51.2 | 114.4 | 137.5 | 122.0 |
| ✓ | 200 | Heun | 50 | – | 215.9 | 462.4 | – | 34.9 | 31.8 | – | 47.8 | 51.1 | – | 132.3 | 124.3 |
| Variant | AdaLN condition | FaceForensics | SkyTimelapse | Taichi-HD | |
|---|---|---|---|---|---|
| baseline | – | 32.9 | 46.6 | 119.1 | |
| w/ | 33.1 | 53.6 | 127.1 | ||
| w/o AdaLN | – | – | 40.6 | 60.8 | 118.2 |
| Variant | AdaLN condition | Depth | Params (M) | FLOPs (T) | Latency (ms) | Peak (GiB) | |
|---|---|---|---|---|---|---|---|
| baseline | – | 28 | 678.8 | 1.057 | 14.55 | 2.678 | |
| w/ | 28 | 678.8 | 1.082 | 15.66 | 2.648 | ||
| w/o AdaLN | – | 28 | 451.4 | 1.081 | 13.59 | 1.794 | |
| w/o AdaLN | – | 42 | 674.6 | 1.624 | 20.12 | 2.650 |
| Method | UCF101 | FaceForensics | SkyTimelapse | Taichi-HD |
|---|---|---|---|---|
| Latte ( Ma et al., 2025 ) | 477.97 | 34.00 | 59.82 | 159.6 |
| Latte+IMG ( Ma et al., 2025 ) | 202.23 | 27.08 | 42.67 | 97.1 |
| FrameDiT+IMG ( Le et al., 2026 ) | 170.1 | 16.6 | 39.5 | 95.5 |
| JVT (Ours) | 136.5 | 32.86 | 46.58 | 119.1 |
| Method | FVD w/ class | FVD w/o class |
|---|---|---|
| AR-Diffusion ( Sun et al., 2025 ) | 186.6 | - |
| Ca2-VDM ( Gao et al., 2025 ) | 184.5 | - |
| ACDiT-XL ( Hu et al., 2026 ) | 111 | - |
| MAGI ( Zhou et al., 2025 ) | - | 297.8 |
| FAR-XL ( Gu et al., 2025 ) | 108 | 279 |
| JVT (Ours) | 136.5 | 215.9 |
| Method | Res. | FVD@10000 |
|---|---|---|
| MAGVIT ( Yu et al., 2023a ) | 76 | |
| MAGVIT-v2 ( Yu et al., 2024 ) | 58 | |
| MaGNeTS ( Goyal et al., 2025 ) | 96.4 | |
| PAR ( Wang et al., 2025b ) | 94.1 | |
| ELT ( Goyal et al., 2026 ) | 60.8 | |
| JVT (Ours) | 91.2 |
Video Diffusion Models
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
See all 783 papers