Direct diffusion modeling of high-dimensional raw video data is challenging, and existing pixel-space methods often rely on specialized architectures. We introduce JVT (Just Video Transformer), a simple large-patch DiT-style Transformer for pixel-space video diffusion, without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net-style skip connections. We systematically examine video patch embeddings, register tokens, and adaptive layer normalization to identify design choices that balance generation quality and computational efficiency. The resulting JVT design achieves generation quality competitive with latent diffusion baselines in both class-conditioned and unconditional settings. The pixel-space formulation of JVT also naturally supports joint generation of pixel-aligned modalities. We demonstrate this capability through camera-conditioned RGB-disparity generation, which improves camera following relative to RGB-only baselines and yields explicit scene geometry without an additional reconstruction model at inference. These results highlight the potential of simple DiT-style Transformers for pixel-space video diffusion, and we hope our findings offer useful insights and practical guidance for future research.
Figures & tables
Figure 1: JVT generates videos directly in pixel space with a simple DiT-style transformer, supporting (a) class-conditioned and (b) unconditional generation. Given an initial RGB frame and a target camera trajectory, it jointly generates (c) RGB frames and disparity, visualized as (d) depth maps. These outputs yield (e) colored point clouds through geometric unprojection, without an additional reconstruction model.
Figure 2: JVT architecture and register token design . The network predicts a clean video x^0 from a noisy input xt . Register tokens are prepended to visual tokens after the first L1 blocks, and then discarded after the remaining L2 blocks, where L=L1+L2 .
Figure 3: Video patch embedding variants (left) and their convolutional components (right) . Assume non-overlapping patches with size pt×ph×pw . T , H , W are divisible by pt , ph , pw , respectively.
Method
Res.
Cond.
Uncond.
UCF-101
UCF-101
FaceForensics
SkyTimelapse
Taichi-HD
MoCoGAN ( Tulyakov et al., 2018 )
2562
–
2886.9 †
124.7 †
206.6 †
–
MoCoGAN-HD ( Tian et al., 2021 )
2562
–
1729.6 †
111.8 †
164.1 †
144.7 ‡
VideoGPT ( Yan et al., 2021 )
2562
–
2880.6 †
185.9 †
222.7 †
–
StyleGAN-V ( Skorokhodov et al., 2022 )
2562
–
1431.0
47.41
79.52
–
MoStGAN-V ( Shen et al., 2023 )
2562
–
1380.3
39.7
65.3
–
Table 1: FVD@2048 across video generation benchmarks. Cond. and Uncond. denote class-conditioned and unconditional generation, respectively. Res. denotes spatial resolution, and all methods use 16-frame clips. Gray scores marked with † / ‡ are taken from StyleGAN-V/DIGAN, and other values are sourced from the original papers. Taichi-HD scores marked with ∗ are trained on both training and test splits, excluded from ranking. JVT with and without “+RG” denote JVT with register guidance and JVT without registers, respectively. – denotes unreported results.
Embedding
Tubelet
Patch-first
Frame-first
Cb
-
32
64
128
16
32
64
8
16
32
FVD ( ↓ )
145.0
147.4
135.6
142.3
143.6
135.0
143.5
151.2
149.7
146.6
Final loss ( ↓ )
0.0151
0.0150
0.0147
0.0143
0.0146
0.0143
0.0141
0.0146
0.0146
0.0142
Denoiser Params (M)
682.2
679.2
679.7
680.8
678.7
678.8
679.0
681.0
683.3
688.1
Denoiser FLOPs (TFLOPs)
1.089
1.083
1.084
1.086
1.082
1.082
1.082
1.086
1.091
1.101
Denoiser Latency (ms, ↓ )
16.07
15.96
15.99
15.58
15.53
15.59
15.59
15.42
15.64
15.89
Table 2: Performance of different video patch embedding variants. Efficiency metrics are measured on one NVIDIA 4090 GPU. Denoiser rows measure the complete model, while Emb. rows measure the isolated video patch embedding module on a 1×16×3×256×256 input. Embedding peak allocated memory includes the input and embedding parameters.
Figure 4: Patch-to-register attention interactions across Transformer blocks. Register tokens are prepended to visual tokens after the first L1=9 blocks.
Guidance
Sampling
FVD@2048 ( ↓ )
CFG
RG
Solver
Steps
NFE
No reg.
er=0
er=ey
er=et
er=ey+et
✓
✗
Euler
50
100
136.5
137.9
138.6
137.2
140.4
✓
✗
Euler
100
200
145.2
135.1
136.8
138.0
142.5
✓
✗
Heun
50
200
167.4
158.7
149.1
157.2
160.8
✓
✓
Euler
50
150
–
129.2
130.4
133.3
132.0
Table 3: Register and sampling ablations for class-conditioned video generation on UCF-101. NFE denotes the number of function evaluations, and No reg. denotes the register-free baseline.
Variant
AdaLN condition
er
FVD ( ↓ )
baseline
et+ey
–
136.5
w/ er=ey
et+ey
ey
138.6
w/o noise level
ey
–
140.6
w/o AdaLN
–
ey
156.3
Table 4: Ablation study of AdaLN on UCF-101. We report class-conditioned FVD@2048. – denotes removal of AdaLN or register tokens.
Figure 5: Joint RGB-disparity generation and point-cloud reconstruction on RealEstate10K . We show generated RGB frames alongside aligned disparity maps. Colored point clouds are obtained by unprojecting depth recovered from disparity using the camera parameters, without an additional reconstruction model at inference.
Method
Objective
FVD ( ↓ )
RRA ( ↑ )
RTA ( ↑ )
AUC ( ↑ )
@5
@30
@5
@30
@5
@30
DFoT ( Song et al., 2025 )
RGB
35.3
99.9
100
77.7
97.4
58.3
87.2
Geometry Forcing ( Wu et al., 2026 )
RGB
46.8
99.8
100
73.6
96.4
52.1
84.7
LIVE ( Huang et al., 2026b )
RGB
42.5
99.1
100
49.8
92.6
28.4
73.3
JVT (Ours)
RGB
27.0
99.9
100
72.4
96.0
52.8
84.1
JVT (Ours)
RGB-disparity
27.5
99.9
100
79.5
98.4
58.6
88.6
Table 5: Camera-conditioned video generation on RealEstate10K. FVD evaluates RGB video quality. Relative rotation accuracy (RRA), relative translation accuracy (RTA), and AUC evaluate agreement between the generated video and the target camera trajectory.
Figure 6: Qualitative comparison of camera-conditioned video generation on RealEstate10K , with four selected frames per scene. GT denotes ground truth and GF denotes Geometry Forcing. JVT and JVT w/ disp. denote the RGB-only and joint RGB-disparity variants, respectively.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Standard RGB
Camera RGB
Camera RGB–disp.
Backbone
JVT-XL
JVT-L
JVT-L, joint I/O
Depth / width / heads
28 / 1,152 / 16
24 / 1,024 / 16
24 / 1,024 / 16
Video size
16×256×256
8×256×256
8×256×256
Input/output channels
3
3
4
Spatial patch size
16×16
Temporal factor
4
1
1
Appendix
Table 6: Model and optimization configurations. The standard-generation column specifies recipes for class-conditioned and unconditional video generation, and the two camera columns summarize the RGB and RGB-disparity configurations. Registers only used for class-conditioned and unconditional video generation on UCF-101. Global batch size is per-rank local batch size multiplied by world size, without gradient accumulation.
Condition
Class-conditioned
Unconditional
Unconditional
Camera control
Dataset
UCF-101
UCF-101
FFS / Sky / Taichi
RealEstate10K
Guidance
CFG or CFG+RG
RG
None
HG
Guidance scale
CFG 4, RG 0.6
0.8
-
4
Solver
Euler
Heun
Heun
Euler
Steps
50
Noise shift s
4
Appendix
Table 7: Sampling configurations for the main results. The camera column covers both the RGB and RGB-disparity configurations. CFG, RG, HG denote classifier-free guidance, register guidance and history guidance, respectively. The register guidance scale is selected by a linear search with a step size of 0.1.
Embedding
Cb
In-context
Tokens
Params (M)
FLOPs
Peak (GiB)
Total (TFLOPs)
Embed. (GFLOPs)
Embed. share (%)
Tubelet
–
yes
1056
682.2
1.089
7.248
0.666
2.674
Tubelet
32
yes
1056
679.2
1.083
1.107
0.102
2.650
Tubelet
64
yes
1056
679.7
1.084
2.215
0.204
2.652
Tubelet
128
yes
1056
680.8
1.086
4.429
0.408
2.659
Patch-first
16
yes
1056
678.7
1.082
0.252
0.023
2.648
Appendix
Table 8: Complete JVT-XL denoiser efficiency results on one NVIDIA 4090 GPU. Embedding FLOPs count the convolution operations in the patch embedder. The embedding share is their percentage of denoiser FLOPs. Peak denotes allocated GPU memory. Cb is the bottleneck width.
Embedding
In-context
Guidance
Fwd.
Latency (ms)
Peak (GiB)
Est. FLOPs (TFLOPs)
Tubelet (plain)
yes
None
1
802.3±1.3
2.689
54.4
Tubelet (plain)
yes
CFG
2
1280.8±1.9
2.751
108.9
Tubelet (plain)
yes
RG
2
1529.9±0.3
2.707
108.9
Tubelet (plain)
yes
CFG+RG
3
2029.3±2.5
2.768
163.3
Tubelet (plain)
no
None
1
739.3±0.8
2.689
53.2
Tubelet (plain)
no
CFG
2
1190.5±2.4
2.751
106.3
Appendix
Table 9: Complete 50-step sampling efficiency results on one NVIDIA 4090 GPU. “Fwd.” is the number of denoiser-forward equivalents per step. Latency is milliseconds per generated 16×256×256 video and is reported as mean ± standard deviation across three sweeps. FLOPs are nominal estimates based on 50 steps and the listed forward equivalents. Register-free branch costs are approximated by the listed configuration’s denoiser cost.
Figure 7: Effect of configured bottleneck width on video patch embedding with batch size 40. FVD uses the left axis and final training loss the right axis. Dashed lines show the corresponding plain-tubelet baseline values. The two vertical scales are independent.
RG
NFE
Solver
Steps
UCF-101
FaceForensics
SkyTimelapse
Taichi-HD
No reg.
er=0
er=et
No reg.
er=0
er=et
No reg.
er=0
er=et
No reg.
er=0
er=et
✗
100
Heun
50
413.5
378.6
474.2
32.9
37.7
33.1
46.6
54.9
52.6
119.1
144.6
127.1
✗
200
Euler
200
421.2
392.4
484.5
35.2
40.2
34.9
49.0
58.3
55.3
114.5
143.6
124.3
✗
200
Heun
100
387.7
354.9
451.5
34.0
39.0
34.0
45.5
54.0
51.2
114.4
137.5
122.0
✓
200
Heun
50
–
215.9
462.4
–
34.9
31.8
–
47.8
51.1
–
132.3
124.3
Appendix
Table 10: Register-token and sampling ablation for unconditional video generation. We report FVD@2048, lower is better. NFE denotes the number of function evaluations, and No reg. denotes the register-free baseline.
Variant
AdaLN condition
er
FaceForensics
SkyTimelapse
Taichi-HD
baseline
et
–
32.9
46.6
119.1
w/ er=et
et
et
33.1
53.6
127.1
w/o AdaLN
–
–
40.6
60.8
118.2
Appendix
Table 11: Effect of timestep conditioning and adaptive layer normalization on unconditional video generation. We report FVD@2048, where lower is better. For the AdaLN condition and er , – denotes the absence of the corresponding conditioning pathway.
Variant
AdaLN condition
er
Depth
Params (M)
FLOPs (T)
Latency (ms)
Peak (GiB)
baseline
et+ey
–
28
678.8
1.057
14.55
2.678
w/ er=ey
et+ey
ey
28
678.8
1.082
15.66
2.648
w/o AdaLN
–
ey
28
451.4
1.081
13.59
1.794
w/o AdaLN
–
ey
42
674.6
1.624
20.12
2.650
Appendix
Table 12: Efficiency comparison of standard JVT and its AdaLN-free variant. We experiment on JVT for class-conditioned task. – denotes removal of AdaLN or register tokens.
Method
UCF101
FaceForensics
SkyTimelapse
Taichi-HD
Latte ( Ma et al., 2025 )
477.97
34.00
59.82
159.6
Latte+IMG ( Ma et al., 2025 )
202.23
27.08
42.67
97.1
FrameDiT+IMG ( Le et al., 2026 )
170.1
16.6
39.5
95.5
JVT (Ours)
136.5
32.86
46.58
119.1
Appendix
Table 13: FVD@2048 on different datasets. We report class-conditioned generation on UCF-101 and unconditional generation on other datasets. “+IMG” means video-image joint training.
Method
FVD w/ class
FVD w/o class
AR-Diffusion ( Sun et al., 2025 )
186.6
-
Ca2-VDM ( Gao et al., 2025 )
184.5
-
ACDiT-XL ( Hu et al., 2026 )
111
-
MAGI ( Zhou et al., 2025 )
-
297.8
FAR-XL ( Gu et al., 2025 )
108
279
JVT (Ours)
136.5
215.9
Appendix
Table 14: FVD@2048 on UCF-101. We compare class-conditioned and unconditional generation with hybrid AR+diffusion methods. All methods use 16×256×256 resolution.
Method
Res.
FVD@10000
MAGVIT ( Yu et al., 2023a )
1282
76
MAGVIT-v2 ( Yu et al., 2024 )
1282
58
MaGNeTS ( Goyal et al., 2025 )
1282
96.4
PAR ( Wang et al., 2025b )
1282
94.1
ELT ( Goyal et al., 2026 )
1282
60.8
JVT (Ours)
2562
91.2
Appendix
Table 15: FVD@10000 on UCF-101. All methods generate 16-frame videos use class condition.
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.