Interactive video-to-video applications require real-time generation while maintaining long-range temporal consistency. However, recent methods achieve speed by restricting attention to a fixed-length causal window. This limits long-range consistency whenever relevant history falls outside the context window. To address this limitation, we introduce vid2vid-long, a correspondence-based approach grounded in a simple observation. Video-to-video translation provides an explicit cross-frame reference: the regions that match in the input video should match in the output video too. Concretely, we compute patch-level correspondences in the input, guiding retrieval of previously generated outputs in the corresponding regions, and condition the model on these patches through correspondence cross-attention. Our experiments demonstrate that vid2vid-long consistently improves long-range temporal consistency across recent video models while preserving generation quality and real-time performance.
Figures & tables
Figure 1: Long-range consistency without long context windows. Standard auto-regressive video translation forgets content once it falls outside its context window (middle row). Our proposed method vid2vid-long (bottom row) explicitly retrieves matching past outputs when the inputs recur, maintaining consistency over long videos. Please see the supplement for a large grid of qualitative result videos.
Figure 2: Correspondence-based video translation. The memory is populated with input-output correspondence pairs from previous generations. To support the consistent translation of the next block of frames, we first retrieve from memory the output patches whose corresponding input is most similar to the current. These output patches, together with the normal input signal, is the input to our video diffusion model. Based on the new induced input-output pairs, we write to memory only those that are sufficiently different from existing.
Family
Method
Base Model
Reconstruction
Quality Score ↑
Efficiency
LPIPS ↓
DreamSim ↓
CLIP-Sim ↑
Latency(s) ↓
FPS ↑
Offline
VACE 1.3B
N/A
0.463
0.195
0.901
0.835
79
1.03
Control-A-Video
SD 1.5
0.288
0.169
0.907
0.722
11
0.89
VideoComposer
N/A
0.664
0.333
0.847
0.724
11
1.36
ControlVideo
SD 1.5
0.174
0.065
0.954
0.778
550
0.59
Auto- regressive
Self-Forcing (4 steps)
VACE 1.3B
0.548
0.203
0.881
0.789
0.67
13.74
Table 1: Quantitative evaluation on depth-to-video translation. We report long-context reconstruction metrics (LPIPS, DreamSim, CLIP-Sim), quality scores (V-Bench), and inference efficiency. Offline methods are shown in gray on the top and distillation based autoregressive methods are shown at the bottom. The best method in each column is marked in bold and the second best method is marked in underline.
Family
Method
Base Model
Reconstruction
Quality Score ↑
Efficiency
LPIPS ↓
DreamSim ↓
CLIP-Sim ↑
Latency(s) ↓
FPS ↑
Offline
VACE 1.3B
N/A
0.592
0.240
0.881
0.819
79
1.01
Control-A-Video
SD 1.5
0.437
0.188
0.895
0.727
14
0.89
VideoComposer
N/A
0.672
0.303
0.854
0.718
12
1.39
ControlVideo
SD 1.5
0.205
0.057
0.957
0.794
2518
0.38
Auto- regressive
Self-Forcing
VACE 1.3B
0.616
0.238
0.870
0.790
0.67
13.74
Table 2: Quantitative evaluation on depth-to-video translation (DL3DV). We report long-context reconstruction metrics (LPIPS, DreamSim, CLIP-Sim), quality scores (V-Bench), and inference efficiency. Offline methods (gray) are shown on the top and distillation based autoregressive methods are in the bottom.The best method in each column is marked in bold and the second best method is marked in underline.
VACE-Bench
DL3DV
LPIPS ↓
DreamSim ↓
CLIP-Sim ↑
Quality Score ↑
LPIPS ↓
DreamSim ↓
CLIP-Sim ↑
Quality Score ↑
Self-Forcing
0.548
0.203
0.881
0.789
0.616
0.238
0.870
0.790
+ Correspondence
0.343 ↓ 37%
0.097 ↓ 52%
0.930 ↑ 6%
0.794
0.451 ↓ 27%
0.151 ↓ 37%
0.897 ↑ 3%
0.775
LongLive
0.511
0.218
0.874
0.817
0.534
0.170
0.896
0.806
+ Correspondence
0.272 ↓ 47%
0.057 ↓ 74%
0.953 ↑ 9%
0.807
0.345 ↓ 35%
0.064 ↓ 63%
0.945 ↑ 5%
0.786
Rolling Forcing
0.427
0.172
0.905
0.799
0.445
0.163
0.907
0.806
Table 3: Sensitivity to the distillation method. Our correspondence mechanism consistently improves long-context reconstruction across all three autoregressive distillation methods, with minimal impact on quality. The base model used is VACE 1.3B.
Figure 3: Qualitative comparisons. We show two frames sampled far apart in time for three example videos (left and right). Bidirectional methods (VACE 1.3B) and Autoregressive distillation methods produce high-quality individual frames but fail to maintain object identity over long durations. Adding our correspondence mechanism preserves consistent appearance throughout the video.
Figure 4: Comparing to alternate consistency mechanisms. We compare different strategies for extending the temporal context beyond sliding window, including enlarging the KV cache, keyframe retention, global pooling, FramePack [ 53 ] compression, and random retrieval. While these approaches provide partial improvements, they suffer for loss of details or quality degradation under long-range revisits. In contrast, our correspondence based method (bottom row) enables accurate reuse of historical content and preserving consistency over long time horizons.
VACE-Bench
DL3DV
Method
LPIPS ↓
DreamSim ↓
CLIP-Sim ↑
Quality ↑
LPIPS ↓
DreamSim ↓
CLIP-Sim ↑
Quality ↑
KV cache ( 2× )
0.350
0.139
0.910
0.795
0.710
0.364
0.814
0.759
KV cache ( 3× )
0.308
0.120
0.920
0.793
0.770
0.496
0.744
0.752
KV cache ( 4× )
0.330
0.134
0.913
0.795
0.759
0.543
0.714
0.753
Keyframe (every 3rd)
0.329
0.115
0.923
0.790
0.607
0.257
0.857
0.776
Keyframe (every 5th)
0.350
0.128
0.914
0.789
0.581
0.240
0.857
0.776
Table 4: Comparison of alternate consistency mechanisms. We compare several strategies for extending temporal context beyond the default sliding window ( W=21 ): increasing the KV cache window size, retaining every N -th keyframe, maintaining a global average-pooled KV cache, Attention Sinks, and FramePack. While larger windows and keyframe retention provide modest reconstruction gains, our correspondence-based retrieval achieves substantially better long-range consistency.
VACE-Bench
DL3DV
Method
LPIPS ↓
DreamSim ↓
CLIP-Sim ↑
Quality ↑
LPIPS ↓
DreamSim ↓
CLIP-Sim ↑
Quality ↑
Ours w/o Corr.
0.379
0.160
0.910
0.829
0.437
0.124
0.921
0.822
Ours
0.240 ↓ 37%
0.066 ↓ 59%
0.955 ↑ 5%
0.828
0.305 ↓ 30%
0.058 ↓ 53%
0.952 ↑ 3%
0.821
Table 5: Importance of correspondence patches. We measure the effect of removing the correspondence patches. Without correspondence, reconstruction metrics degrade substantially while quality scores remain similar, demonstrating the effectiveness of correspondence tokens for maintaining long term consistency.
Figure 5: Effect of correspondence pointers on long-range consistency. Without correspondence pointers, the model generates plausible but inconsistent frames. Object appearance drifts over time. Consider the top example, the dog starts off having a yellow snout, but it changes to blue when it re-appears. With correspondence pointers, the model retrieves and reuses visual context from earlier frames, maintaining consistent identity throughout the video. For example in the top video, the dog snout is yellow at the beginning of the video. However it turns into a blue snout when reappearing if correspondence is not used. However it stays consistently yellow with our method.
Figure 6: Network architecture. We show a single transformer block at layer ℓ on the left. Our method inserts a new correspondence cross-attention layer between existing self-attention and text cross-attention layers. Details of the correspondence cross-attention are shown on the right. For each token at index g , the current hidden state xg serves as the query and is projected to query Qg . The k retrieved neighbors are projected to keys Kg and values Vg (see Section 3.3 of main paper). The standard scaled dot-product attention produces the output Ag , which is projected through Wo and multiplied by the binary mask mg . The result is added back to xg as a residual connection.
Figure 7: Per-component runtime breakdown. Inference time for generating a 20-second video (320 frames) on a single H100 GPU at 480p, for each autoregressive backbone with our correspondence mechanism. The denoising forward passes dominate the total runtime across all methods. Hatched regions indicate the correspondence cross-attention overhead within the denoising and KV cache components. The inset zooms into the memory read/write component, which is negligible ( <50 ms) across all methods.
480p (480 × 832)
720p (720 × 1280)
Duration
Frames
Full KV
Ours
Full KV
Ours
∣H∣
10 s
160
42.2
34.1
63.7
45.2
2 K
30 s
480
74.3
34.1
OOM
45.2
4 K
1 min
960
OOM
34.1
OOM
45.2
6 K
2 min
1920
OOM
34.1
OOM
45.3
9 K
5 min
4800
OOM
34.2
OOM
45.4
16 K
Table 6: VRAM scaling with video length. Full KV-cache retains all past tokens across all transformer layers, causing memory to grow linearly with video length. Our method replaces the unbounded cache with a fixed kv cache plus a lightweight correspondence memory. Total VRAM remains nearly constant regardless of video duration at both resolutions. OOM denotes exceeding the 80 GB capacity of an H100 GPU.
Figure 8: Effect of history compression. Left: History size ∣H∣ vs. video length (log scale). Without compression, the history grows linearly to over 936 K entries at 9 600 frames (10 minutes). With our redundancy-based write policy ( τh=0.75 ), the history grows sub-linearly to ∼ 26 K entries ( ∼36× reduction). Right: Corresponding per-block memory read and write time. Without compression, the cost grows linearly, exceeding 30 ms at 10 minutes. With compression, it remains below 2 ms throughout. Shaded regions show ± 1 standard error across 30 videos spanning three categories (webcam, surveillance, action camera).
τh
∣H∣
LPIPS ↓
CLIP ↑
DreamSim ↓
0.25
413
0.301
0.940
0.092
0.50
682
0.284
0.946
0.081
0.75
3,897
0.256
0.957
0.060
0.90
15,822
0.204
0.963
0.049
Table 7: Hyperparameter analysis. We show the effects of varying the thresholds τh and τm on our method. As the threshold τh is increased, our method stores more patches in the history bank H and improves the reconstruction performance.
Method
LPIPS ↓
CLIP ↑
DreamSim ↓
Self-Forcing
0.474
0.906
0.167
LongLive
0.384
0.911
0.156
Rolling Forcing
0.284
0.940
0.100
Ours
0.191
0.960
0.050
Table 8: Other conditioning inputs We show evaluation with sketch inputs (left) and optical flow inputs (right).
Hyperparameter
Value
Student model
VACE 1.3B
Teacher model
VACE 14B
Distillation method
Rolling Forcing
Distillation loss
DMD
Denoising steps
5
Guidance scale
5.0
Table 9: Training and model hyperparameters.
VACE-Bench
DL3DV
Feature Extractor ϕ
LPIPS ↓
DreamSim ↓
CLIP-Sim ↑
Quality ↑
LPIPS ↓
DreamSim ↓
CLIP-Sim ↑
Quality ↑
VAE latent
0.240
0.066
0.955
0.828
0.305
0.058
0.952
0.821
DINOv2 [ 27 ]
0.336
0.122
0.926
0.812
0.296
0.068
0.943
0.798
CLIP [ 33 ]
0.297
0.107
0.931
0.821
0.288
0.057
0.952
0.810
Table 10: Ablation of the feature extractor ϕ . We compare three choices for computing patch descriptors: VAE latent features, DINOv2, and CLIP. VAE latent features achieve the best reconstruction on VACE-Bench and competitive performance on DL3DV, without requiring an external feature model. All variants use k=3 retrieved neighbors.
Feature Extractor ϕ
VAE Decode
Backbone
Total (20s video)
VAE latent
—
—
0.3 ms
DINOv2 [ 27 ]
17.4 ms
62.2 ms
2.3 s
CLIP [ 33 ]
17.4 ms
14.7 ms
0.9 s
Table 11: Runtime overhead of different feature extractors. Using DINOv2 or CLIP requires decoding each block back to RGB and running the vision backbone, adding significant per-block overhead. VAE latent features avoid this cost entirely by reusing the already-available encoder output. Measured on a single A100 GPU at 480p resolution.
VACE-Bench
Subj. ↑
BG ↑
Flicker ↑
Motion ↑
Aesth. ↑
Imaging ↑
Avg. ↑
Offline
VACE 1.3B
0.929
0.935
0.934
0.962
0.606
0.643
0.835
Control-A-Video
0.668
0.799
0.875
0.878
0.458
0.657
0.722
VideoComposer
0.761
0.864
0.853
0.852
0.475
0.537
0.724
ControlVideo
0.799
0.867
0.911
0.927
0.573
0.591
0.778
Auto- regressive
Self-Forcing (4 steps)
0.779
0.835
0.914
0.944
0.623
0.640
0.789
Table 12: Per-metric V-Bench quality scores. We report individual normalized V-Bench [ 13 ] metrics for all methods on both benchmarks. The rightmost column shows the average across all six metrics (reported as the aggregate quality score in the main paper). Best among autoregressive methods is bold , second best is underlined .
VACE-Bench
Subj. ↑
BG ↑
Flicker ↑
Motion ↑
Aesth. ↑
Imaging ↑
Avg. ↑
Self-Forcing
0.779
0.835
0.914
0.944
0.623
0.640
0.789
+ Correspondence
0.849
0.877
0.912
0.941
0.601
0.581
0.794
LongLive
0.873
0.891
0.916
0.953
0.619
0.650
0.817
+ Correspondence
0.875
0.894
0.906
0.935
0.603
0.628
0.807
Rolling Forcing
0.819
0.863
0.936
0.962
0.597
0.616
0.799
Table 13: Per-metric V-Bench quality scores for each distillation method. We report individual normalized V-Bench [ 13 ] metrics for each autoregressive distillation backbone with and without our correspondence mechanism. Adding correspondences maintains comparable overall visual quality across different autoregressive backbones, with some variations in individual metrics.