JtT: Rethinking Point Trajectory Diffusion for Diverse and Rigid Motion from an Image

Gabrijel Boduljak; Yushi Lan; Christian Rupprecht; Andrea Vedaldi

cs.CVOct 12, 2026

JtT: Rethinking Point Trajectory Diffusion for Diverse and Rigid Motion from an Image

Authors: Gabrijel Boduljak, Yushi Lan, Christian Rupprecht, Andrea Vedaldi

Organizations: Visual Geometry Group, University of Oxford

Abstract

We study forecasting rigid-body motion from a single image: generating diverse and physically plausible future trajectories for arbitrary points in a scene, without building rigidity into the architecture or relying on geometric inputs such as depth. Like video generators, recent trajectory generators fail at this task, even in simple synthetic Kubric scenes, despite scaling up model size and training on real data. We show how to solve this with the right architecture and parameterization. Our model, JtT, is a simple diffusion transformer that operates directly in coordinate space and decodes arbitrary and dense query patterns, without the trajectory VAEs or fixed grids assumed in prior work. Through a systematic study of the design space of point trajectory diffusion, we find that diffusing per-frame velocities is the only parameterization that captures rigidity, that it must be paired with the right timestep schedule, and that adding full-resolution RGB at the query points resolves the boundary failures of prior methods. However, evaluating such 2D models rigorously is itself non-trivial. To our knowledge, we are the first to observe that physical plausibility and diversity do not improve together during training: larger models, as used in prior work, memorize the single ground-truth future per training image, becoming more rigid while covering fewer possible futures. Yet prior work has paid little attention to this, so we show how model rankings depend on model size, metric, and stopping time, and adopt a stricter, checkpoint-averaged evaluation protocol. Under this protocol, JtT improves over prior methods on both distributional and rigidity metrics and approaches ground-truth rigidity, with only about 50M parameters.

Figures & tables

Figure 1: Our method, JtT , infers diverse, physically plausible motion for an arbitrary number of query points, supporting flexible query patterns across diverse camera viewpoints and scene settings.

Method	Queries	Dense	Space	Target	Image conditioning	Parameters
	Flexibility		Representation
Track2Act	arbitrary	✗	coords.	absolute $x_{t}$	global vector	600M
WHN	fixed $32{\times}32$	✗	VAE latent	absolute $x_{t}$	DINOv2 patches	100/250M
TrajLoom	fixed $15{\times}26$	✗	VAE latent	offset $x_{t}-x_{c}$	Wan2.1 latents	250M
MotionForecasting	arbitrary	✗	coords.	velocity $\Delta_{t}$	DINOv3 at query	100M
GPTF	arbitrary	✗	coords.	absolute $x_{t}$	DINOv3 patches	30/230M ∗

Table 1: Design space of image-conditioned trajectory generators. Queries : where trajectories can be requested at inference; Dense : decodes grids denser than in training; Space : where diffusion is performed; Target : the diffused trajectory representation; ∗ : estimated ( Section A.3 ).

Figure 2: Overview of JtT . Given an input image and query points, image patch tokens ( Tg gray ) are formed by concatenating patch position $(x,y)$ , RGB content, and multi-layer DINOv2 features ( $l\in\{3,6,9,12\}$ ), while query points are tokenized analogously with position, sampled RGB, and noised per-frame velocities $(\Delta_{t})_{t=1}^{T}$ , and then split into a support set ( Tg cyan ) and a non-support set ( Tg pink ). A Read-To-Support cross-attention block initializes support tokens from non-support tokens. The Support Encoder ( $\times N$ ) refines the support set via interleaved self-attention and cross-attention to image patches. The Non-Support Decoder ( $\times M$ ) reads out from the encoded support set in parallel; non-support tokens never attend to each other or back to the image, making dense readout linear in $M$ and reusable for any query pattern at inference. A linear Velocity Head predicts the rectified-flow velocities, which are accumulated to recover trajectories. All blocks are AdaLN-modulated by the diffusion timestep $\tau$ .

Panel of Figure 3 — Figure 3: Evolution of downstream metrics and flow-matching loss during baseline training (Kubric-A; lower is better). Larger models attain their best FVMD early and then memorize, as their validation flow-matching loss (dashed) rises while the training loss (solid) keeps falling; dotted lines mark each model’s lowest validation loss (thin: checkpoints; thick: 3-checkpoint moving average).

Model	Size	FVMD (D) $\downarrow$	FVMD (S) $\downarrow$	Best-of- $K$ $\downarrow$	LRTL (ALL) $\downarrow$
GT	—	4,941	8,845	110.85	6.68
Track2Act (from WHN)	600M	16,735	22,509	250.80	15.80
WHN (L, latent-space)	250M	13,745	17,838	127.00	14.10
WHN (B, coordinate-space)	50M	14,519	18,551	137.80	23.70
MotionForecasting †	100M	20,178	26,379	186.47	8.20
GPTF	30M ∗	—	17,786	95.60	14.60

Table 2: Motion generation quality on Kubric-A, following WHN. Prior methods: single checkpoint; JtT : Average (Best) of the last five checkpoints. JtT improves over prior methods on FVMD and LRTL, nearing the ground truth; GPTF is best on Best-of-

K

. † : reimplementation; ∗ : estimated.

Figure 4: Qualitative results on Kubric from WHN ( Boduljak et al., 2026 ) . Baselines often miss smaller objects ( Tg red ) or object parts ( Tg dark green ), and frequently fail to capture boundary points accurately ( Tg peach ), primarily due to their low-resolution conditioning. In contrast, our method reliably captures objects and their parts while achieving near-perfect boundary accuracy. This highlights our token design, which uses full-resolution RGB at the query points.

$\tau\sim\operatorname{Uniform}[0,1]$
Model	FVMD (D)	FVMD (S)	Best-of- $K$	LRTL (ALL)	LRTL (FG)
absolute	18,695 (15,750)	23,719 (20,772)	153.54 (149.67)	14.76 (11.65)	13.22 (11.56)
velocities	19,927 (17,785)	25,914 (23,518)	125.31 (122.19)	5.12 (4.81)	5.38 (5.17)
offsets	13,915 (13,164)	19,100 (18,245)	133.68 (124.97)	9.74 (8.61)	9.90 (8.93)
$\tau\sim\operatorname{LogitNormal}(0,1)$
absolute	20,284 (19,140)	24,678 (23,801)	125.15 (118.90)	25.59 (21.87)	19.25 (17.16)

Table 3: The impact of diffusion target parameterization and timestep scheduling. Relative parameterizations outperform absolute coordinates; diffusing velocities with logit-normal timestep sampling gives the best overall trade-off between diversity and rigidity.

Kubric-A
Model	FVMD (D)	FVMD (S)	Best-of- $K$	LRTL (ALL)	LRTL (FG)
2D	14,097 (12,080)	20,074 (17,790)	120.04 (111.50)	6.16 (5.75)	5.92 (5.70)
2D + Inverse Depth	13,946 (12,560)	19,827 (17,962)	92.51 (83.98)	5.59 (4.21)	5.97 (4.77)
3D	12,909 (12,162)	18,518 (17,641)	95.02 (92.37)	4.37 (3.86)	4.82 (4.38)
Kubric-D
2D	19,210 (17,848)	28,412 (25,903)	94.16 (89.68)	2.05 (1.95)	1.69 (1.60)

Table 4: Explicit oracle geometry helps. Inverse depth or 3D world coordinates mainly improve Best-of-

K

, while in rigidity our 2D-only model closes most of the gap from prior 2D methods to the 3D oracle. Kubric-A/D are re-rendered with 2D–3D correspondence, so numbers differ from Table 2 .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Token with	FVMD (D)	FVMD (S)	Best-of- $K$	LRTL (ALL)	LRTL (FG)
Nothing	13,458 (13,006)	17,662 (17,147)	182.73 (175.60)	8.49 (7.61)	5.95 (5.56)
DINO	14,470 (13,901)	19,189 (18,357)	184.19 (170.87)	8.94 (8.48)	6.15 (5.92)
RGB	14,136 (12,386)	18,523 (16,589)	138.27 (122.00)	6.82 (6.43)	6.35 (6.10)
Inverse Depth	15,572 (13,653)	20,116 (17,716)	88.72 (81.63)	4.11 (3.82)	4.56 (4.30)
Segm (binary)	13,232 (12,753)	17,281 (16,816)	93.49 (86.18)	5.84 (5.29)	6.58 (5.99)
Segm (instance)	13,356 (12,888)	17,666 (17,205)	104.64 (97.77)	6.01 (5.24)	6.77 (5.93)

Appendix

Table 5: Impact of full-resolution inputs on motion generation. RGB alone yields strong results, yet more explicit representations, such as segmentation and inverse depth, further improve performance.

Model	FVMD (D)	FVMD (S)	Best-of- $K$	LRTL (ALL)	LRTL (FG)
B	13,514 (11,812)	17,716 (15,829)	124.37 (115.24)	6.67 (6.13)	6.01 (5.58)
B (Dense)	13,266 (12,333)	17,437 (15,958)	130.44 (118.76)	6.80 (6.13)	5.87 (5.31)
S	12,546 (12,238)	16,769 (16,351)	160.25 (129.77)	7.69 (6.91)	6.57 (6.26)
S (Dense)	11,807 (11,591)	15,695 (15,452)	135.85 (128.69)	6.84 (6.64)	5.82 (5.46)

Appendix

Table 6: Adding a support-point architecture. Besides enabling dense decoding, the support-point architecture (Dense) improves FVMD and foreground rigidity (LRTL (FG)) over the baseline (Sparse) for both model sizes, and improves every metric for S. The slight degradation in Best-of-

K

and LRTL (ALL) for B is expected, as the Dense model solves a harder task, and is not visible qualitatively.

Feature extractor	FVMD (D)	FVMD (S)	Best-of- $K$	LRTL (ALL)	LRTL (FG)
DINOv2 S	14,136 (12,386)	18,523 (16,589)	138.27 (122.00)	6.82 (6.43)	6.35 (6.10)
DINOv2 S+Reg	13,441 (12,793)	17,623 (16,763)	126.00 (116.75)	6.67 (5.95)	6.21 (5.86)
DINOv2 S+Reg+DPT	13,514 (11,812)	17,716 (15,829)	124.37 (115.24)	6.67 (6.13)	6.01 (5.58)
DINOv2 G+Reg+DPT	14,519 (12,522)	18,482 (16,442)	123.72 (104.64)	6.75 (5.88)	6.04 (5.57)
4RC+DPT	13,363 (12,227)	17,441 (16,284)	128.20 (121.22)	6.40 (5.56)	5.88 (5.29)
VGGT-Omega+DPT	13,821 (12,824)	18,045 (17,054)	150.18 (125.42)	7.27 (6.43)	6.53 (6.09)

Appendix

Table 7: The impact of the image feature extractor on motion quality on Kubric-A. All models diffuse velocities; only the feature extractor changes. Adding register tokens (Reg) improves all metrics, and multi-layer (DPT) features further improve Best-of-

K

and foreground rigidity. Larger or alternatively pretrained extractors (DINOv2 G, 4RC, VGGT-Omega, VJEPA2.1) do not yield consistent gains. Results are reported as Average (Best).

Sampler	NFE	FVMD (D)	FVMD (S)	Best-of- $K$	LRTL (ALL)	LRTL (FG)
ODE	10	11,614	15,263	115.59	7.05	6.41
	25	11,404	15,221	126.82	7.95	7.18
	50	11,365	15,266	133.83	8.35	7.51
	100	11,357	15,311	138.16	8.59	7.71
	250	11,359	15,348	141.45	8.74	7.84
	1000	11,362	15,370	143.29	8.83	7.90

Appendix

Table 8: Effect of the sampler and number of function evaluations (NFE) on Kubric-A, for a single JtT -B (Sparse) checkpoint at 70K steps. Our default is the 1000-step SDE (Euler–Maruyama) sampler.

Panel of Figure 5 — Figure 5: Distributional fidelity during training (Kubric-A; lower is better), enlarged from Fig. 3 . B and S reach their best FVMD at 50K and 70K steps, respectively, and then degrade as they memorize the single ground-truth future per training image, whereas the FVMD of XS decreases steadily. Thin lines: evaluated checkpoints; thick lines: 3-checkpoint moving average.

Panel of Figure 6 — Figure 6: Best-of- $K$ and rigidity during training (Kubric-A; lower is better), complementing Fig. 3 . Best-of- $K$ and LRTL keep improving for S and B even after their FVMD degrades, as their samples collapse toward the single ground-truth future per training image. Thin lines: evaluated checkpoints; thick lines: 3-checkpoint moving average.

Panel of Figure 7 — Figure 7: Flow-matching loss over the full training run (Kubric-A; log scale), averaged over diffusion timesteps 0.1–0.9, complementing Fig. 3 . Solid: training loss; dashed: validation loss. The validation loss of B is lowest at 60K steps and of S at 100K steps, after which both rise while their training losses keep falling; for XS, the two losses stay close and still decrease at 200K steps.

Model	Step	FVMD (D)	FVMD (S)	Best-of- $K$	LRTL (ALL)	LRTL (FG)
XS	40K	18,319	21,505	165.15	7.38	5.76
	50K	14,165	18,125	154.48	8.71	7.28
	60K	13,092	17,539	189.52	10.85	9.36
	70K	13,020	17,629	201.34	11.60	10.16
	80K	12,976	17,533	197.07	11.17	9.97
	90K	13,004	17,596	202.53	11.20	9.97

Appendix

Table 9: Model performance through training for different model sizes on Kubric-A, up to 130K steps. Larger models reach their best FVMD earlier (B at 50K, S at 70K steps), after which FVMD increases while Best-of-

K

and LRTL keep improving; the FVMD of XS decreases almost monotonically.

Model	Step	FVMD (D)	FVMD (S)	Best-of- $K$	LRTL (ALL)	LRTL (FG)
XS	50K	14,165	18,125	154.48	8.71	7.28
	100K	12,644	17,118	186.86	10.35	9.36
	150K	11,737	15,873	130.47	7.21	7.12
	200K	11,771	15,773	120.78	6.52	6.68

Appendix

Table 10: Convergence of JtT -XS (under 10M parameters) trained for 200K steps on Kubric-A.

Model	Step	FVMD (D)	FVMD (S)	Best-of- $K$	LRTL (ALL)	LRTL (FG)
S	60000	13068.82	17512.31	204.86	9.27	7.41
	70000	12237.69	16350.72	169.90	7.54	6.26
	80000	12328.60	16447.45	155.07	7.51	6.46
	90000	12746.66	16941.53	141.66	7.23	6.45
	100000	12349.56	16597.45	129.77	6.91	6.30
S (Dense)	60000	11957.53	15733.35	137.26	6.64	5.46

Appendix

Table 11: Per-checkpoint results of the Sparse and Dense models on Kubric-A, from which the Average (Best) values in Tables 2 and 6 are computed. Several metrics are non-monotonic across checkpoints.

Model Variant	Enc. Depth	Dec. Depth	Hidden Size ( $d_{model}$ )	Heads	Head Dim
JtT -XS	6	2	192	3	64
JtT -S	12	2	384	6	64
JtT -B	12	4	768	12	64

Appendix

Table 12: JtT model variants.

Parameter	Default	Description
traj_length	24	Number of frames/timesteps in the trajectory sequence.
in_channels	2	Input coordinate dimensions (e.g., $x$ and $y$ ).
frame_feature_size	384	Dimensionality of the pre-extracted vision features.
frame_feature_patch_size	14	Spatial patch size used by the vision backbone.
mlp_ratio	4.0	Expansion factor for the hidden dimension in the MLP blocks.
num_freq_bands	8	Number of frequency bands for Fourier/Sin-Cos featurization.

Appendix

Table 13: Shared architecture hyperparameters.

	Parameter	Value
Model	Stochastic Interpolant Path	Linear (Rectified-Flow)
	Stochastic Interpolant Target	velocity
	Network Prediction	velocity
	Parameterization	velocities $\Delta_{t}$
	Feature Extractor	DINOv2
	Feature Extractor Size	Small

Appendix

Table 14: Hyperparameters for JtT .

Figure 8: Comparison on the Kubric benchmark of WHN ( Boduljak et al., 2026 ) . Scene 42.

Figure 9: Comparison on the Kubric benchmark of WHN ( Boduljak et al., 2026 ) . Scene 45.

Figure 10: Comparison on the Kubric benchmark of WHN ( Boduljak et al., 2026 ) . Scene 48.

Figure 11: Comparison on the Kubric benchmark of WHN ( Boduljak et al., 2026 ) . Scene 52.

Figure 12: Comparison on the Kubric benchmark of WHN ( Boduljak et al., 2026 ) . Scene 57.

Figure 13: Qualitative results on Physics 101 ( Wu et al., 2016 ) , following the training and evaluation setup of WHN ( Boduljak et al., 2026 ) .

Figure 14: More qualitative results on Physics 101 ( Wu et al., 2016 ) , following the training and evaluation setup of WHN ( Boduljak et al., 2026 ) .

Figure 15: More qualitative results on Physics 101 ( Wu et al., 2016 ) , following the training and evaluation setup of WHN ( Boduljak et al., 2026 ) .

Figure 16: Sample diversity on Physics 101 ( Wu et al., 2016 ) . Four samples from JtT per input frame, following the training and evaluation setup of WHN ( Boduljak et al., 2026 ) .

Figure 17: Qualitative results on Cityscapes ( Cordts et al., 2016 ) , using the training and evaluation split of WHN ( Boduljak et al., 2026 ) .

Figure 18: More qualitative results on Cityscapes ( Cordts et al., 2016 ) , using the training and evaluation split of WHN ( Boduljak et al., 2026 ) .

Figure 19: Sample diversity on Cityscapes ( Cordts et al., 2016 ) . Three samples from JtT per input frame, using the training and evaluation split of WHN ( Boduljak et al., 2026 ) .

Trajectory Prediction

Trajectory prediction forecasts how people, vehicles, robots, and other dynamic systems will move from observed histories and context.

See all 187 papers

RigidBench: Evaluating Rigid-Body Physics in Video Generation Models
Aug 16, 2026Swarnim Jain, Shangzhe WuPhysical Consistency in Video Generation Video Generation Evaluation
University of Cambridge
TrajLoom: Dense Future Trajectory Generation from Video
Mar 23, 2026Zewei Zhang, Jia Jun Cheng Xian, Kaiwen Liu +4Trajectory Prediction Video Prediction
McMaster University · University of British Columbia · Vector Institute +2
PhysiFormer: Learning to Simulate Mechanics in World Space
Jun 25, 2026Yiming Chen, Yushi Lan, Andrea VedaldiWorld Model Learning Physics-Based Simulation
Visual Geometry Group, University of Oxford

Aug 16, 2026J/K move · Enter open · S save

RigidBench: Evaluating Rigid-Body Physics in Video Generation Models

Swarnim Jain, Shangzhe Wu

University of Cambridge

Physical Consistency in Video Generation Video Generation Evaluation Video Diffusion Models

Open paper AskPDF arXiv

Video models are increasingly used to predict what happens next in a scene, yet the metrics commonly used to compare their outputs say little about whether the predicted objects move correctly. Motion, geometry, identity, background stability, and visual similarity can fail independently, but whole-frame scores often mix these errors together. We introduce RigidBench, a simulator-grounded benchmark that compares a generated continuation with a reference rollout from the same initial frame and motion description. Its five rigid-body tasks vary objects, materials, viewpoints, and indoor and outdoor scenes, with per-frame masks, depth, 6-DoF trajectories, and contacts available for scoring. We evaluate eight models on the same 100 examples with ten measurements that keep these aspects separate. The resulting rankings depend strongly on what is measured: no model leads on all ten, and across model means, higher SSIM accompanies larger 3D trajectory error (r = 0.89). RigidBench also includes 5,000 training videos with exact simulator state, which we use to fine-tune and analyze Wan 2.2 TI2V-5B. Full fine-tuning reduces 3D trajectory error by about 20% with almost no change in SSIM, while teacher-forced probes and targeted interventions show that object position is represented throughout Wan's diffusion transformer and used by its denoising computation.

Abstract

Trajectory Prediction

Explore similar work

RigidBench: Evaluating Rigid-Body Physics in Video Generation Models

TrajLoom: Dense Future Trajectory Generation from Video

PhysiFormer: Learning to Simulate Mechanics in World Space