Step-by-step painting tutorials are used for learning artistic techniques, but existing video resources lack interactivity and personalization. While recent generative models have advanced artistic image synthesis, they still struggle to generate temporally and structurally consistent videos. Video models excel in consistent frame generation, yet fine-tuning an image-to-video model on forward painting sequences creates a long-range dependency on the first-frame reference, disrupting its pretrained attention pattern and hindering reconstruction. We address this issue by training on reversed sequences that better preserve the reference-attention behavior. Building on this, we present Loomis Painter, a framework for reconstructing coherent painting processes. We further introduce a media-transfer training strategy that decouples input appearance from the target process, enabling generation across different artistic media. Trained on a curated dataset of real-world painting processes, our method outperforms existing baselines across LPIPS, DINO, and CLIP metrics and is consistently preferred in expert and non-expert user studies. Code available at https://github.com/Markus-Pobitzer/wlp.
Figures & tables
(a) Loomis-style pencil portraits painting from any portrait photo.
(b) Reconstructing the painting process while preserving the input image.
Figure 1: Overview of our painting process generation method. The curated video is first reversed to better align with the I2V model. A video generation model conditioned on an input image and a prompt is LoRA-tuned. The input image is a painting, and the model learns to reconstruct the steps to paint it, starting from a finished painting to a blank canvas. In (b), the media transfer model is shown, which enables to render any input image as an acrylic, oil, or pencil painting based on the text input. To achieve this, we generate variations of the reference image using image editing models and train the video generation model to reconstruct the original painting process.
Figure 2: Comparison of final frame. On the left, the forward process is not able to reconstruct the input image fully; details are still missing, i.e., bottom right part. Image courtesy Samir Godinjak, from the Painting with Samir YouTube channel.
Figure 3: Reference attention distribution (WAN I2V 5B). Probability of attending to the reference frame relative to non-self frames ( P(attend to ref.∣not self) ). Our reverse painting model maintains a decaying attention profile similar to the base model, whereas forward painting shifts focus heavily toward later frames. Fine-tuned models were trained for 7 epochs. Stepwise plateaus of 4 frames reflect the model’s 4× temporal latent compression. See Fig. 8 in the appendix for additional architectures.
Figure 4: Dataset Curation Pipeline Overview. Our framework extracts painting workflows from raw tutorial videos. First, start and end frames are detected, and the painting canvas is localized. The video is then partitioned into N segments, from which M frames are sampled per segment. Subsequently, occlusions (e.g., hands, brushes) are detected in the sampled frames, and a masked median is computed over the sampled frames, using the preceding median frame as a reference to remove transient obstructions. Producing N occlusion-free frames. Image courtesy of Samir Godinjak, from the Painting with Samir YouTube channel.
Acrylic
Oil
Pencil
Loomis
Total
# of Videos
81
151
298
207
737
Avg. Duration [min]
≈ 40
≈ 30
≈ 12
≈ 20
—
Table 1: Overview of the curated video dataset by art medium.
Figure 5: User preference evaluation (N=22). Blue bars indicate a preference for our method, red bars indicate a preference for the baseline, and the yellow center represents a tie. Users preferred our results over ProcessPainter, Inverse Painting, and PaintsUndo in 96.2%, 78.7%, and 95.6% of comparisons, respectively.
Figure 6: Sampled frames of our method with input images. Painting order from left to right, top to bottom. The image with the green border is the reference painting. (a) Illustrates a typical painting process where the model first creates a sketch of the mountain, paints the background, layers the colors of the foreground objects, and finishes with fine details. (b) and (d) Show pencil paintings where outlines are sketched first, followed by details and depth added via hatching. (c) An acrylic painting process where the scene gets first sketched with blue color before being painted, the input image is a pencil sketch. (e) Demonstrates the Loomis portrait method, which uses simple circles and division lines to faithfully recreate the head.
Method
FID ↓
LPIPS ↓
CLIP ↑
DINOv2 ↑
DDC ↓
Latency (s)
FPS
Params
Inverse Painting [ 9 ]
326.15
0.61
0.66
0.21
48.77
333.8
0.16
900M
ProcessPainter [ 47 ]
282.90
0.53
0.76
0.50
17.79
1.7
4.71
1.3B
PaintsUndo [ 58 ]
236.52
0.55
0.77
0.56
15.05
129.6
0.62
900M
PaintsUndo † [ 58 ]
196.66
0.43
0.83
0.66
10.02
129.6
0.62
900M
Ours (VACE)
183.61
0.42
0.83
0.70
9.18
171.7
0.47
1.3B
Ours (Wan 2.2)
152.42
0.38
0.85
0.75
7.97
37.3
2.17
5B
Table 2: Evaluation using various metrics for measuring fidelity and inference speed. The number of parameters only relates to the image/video generation component. † denotes fine-tuned on our dataset. Best overall results in bold, best baseline results underlined.
Method ∣ Video
1
2
3
4
5
6
Avg.
Stroke Coherence
PaintsUndo
2.4
1.5
1.6
2.0
1.6
1.0
1.7±1.1
Ours
3.4
4.2
3.4
4.0
3.8
4.3
3.9±1.0
GT
4.2
4.8
4.8
4.4
4.0
5.0
4.5±0.9
Layering Logic
PaintsUndo
2.2
1.5
1.1
2.2
1.6
1.1
1.7±1.1
Table 3: Expert evaluation results. Participants rated videos across five metrics on a 5-point Likert scale. Our method consistently outperforms the baseline, approaching Ground Truth (GT) in several criteria.
Forward
Reverse (Ours)
Model
Epoch
FID ↓
LPIPS ↓
FID ↓
LPIPS ↓
5B
1
171
0.414
187
0.436
5B
3
179
0.424
166
0.395
5B
5
175
0.424
160
0.389
5B
7
163
0.401
159
0.390
14B
7
173
0.417
164
0.393
Table 4: Ablation study evaluating the impact of our proposed reverse painting training strategy. Comparing the base video generation model against our method under the same training constraints demonstrates that the reverse painting strategy converges faster and improves visual fidelity.
Figure 7: Method comparison. Painting order from left to right. The final frame (green border) is visually nearly indistinguish- able from the reference image; the latter is omitted for brevity.
Figure 8: Reference attention distribution. Probability of attending to the reference frame relative to non-self frames ( P(attend to ref.∣not self) ). Our reverse painting model maintains a decaying attention profile similar to the base model across architectures, whereas forward painting shifts focus heavily toward later frames. Fine-tuned models were trained for 7 epochs. Stepwise plateaus of 4 frames reflect the model’s 4× temporal latent compression.
Method
FID ↓
LPIPS ↓
CLIP ↑
DINOv2 ↑
DDC ↓
Distance
Params
LPIPS ↓
CLIP ↑
DINOv2 ↑
Qwen + Base
306.03
0.70
0.74
0.52
41.16
0.68
0.86
0.79
20B + 5B
Fine tuned + Base
169.32
0.45
0.84
0.73
11.97
0.24
0.96
0.97
5B + 5B
Art Media Transfer
168.78
0.44
0.84
0.73
10.36
0.22
0.97
0.97
5B
Oracle + Base
152.42
0.38
0.85
0.75
7.97
0.03
0.98
0.99
5B
Table 5: Quantitative comparison of our proposed Art Media Transfer model against baselines. Best non-oracle results are in bold . Art Media Transfer achieves the highest overall performance while requiring significantly fewer parameters. “Distance” measures visual similarity between the generated final frame and the ground truth. Oracle + Base establishes the theoretical upper bound.
Figure 9: Screenshot of the expert user study. These questions were asked for each painting video.
Model
Params
Peak VRAM (GB)
Latency (s)
FPS
Wan 2.1
14B
47.0
428.3
0.19
Wan 2.2
5B
35.3
37.3
2.17
Vace
1.3B
18.4
171.7
0.47
Table 6: Inference Efficiency Comparison. Benchmarked on a single NVIDIA A100 GPU generating 81 frames at 832×480 spatial resolution in bf16 precision. Best results per column are highlighted in bold .
Method A
Method B
Win (A) (%)
Tie (%)
Win (B) (%)
Total
p -value
Ours
PaintsUndo
95.6
1.3
3.2
158
<0.001
Ours
Inverse Painting
78.7
9.8
11.6
164
<0.001
Ours
ProcessPainter
96.2
0.0
3.8
160
<0.001
PaintsUndo
Inverse Painting
27.2
14.2
58.6
162
<0.001
PaintsUndo
ProcessPainter
47.0
14.6
38.4
164
0.272
Inverse Painting
ProcessPainter
63.4
8.5
28.0
164
<0.001
Table 7: Human evaluation results for overall preference. Method A and Method B denote the two compared approaches. The p -values indicate statistical significance.
Top Performing Images
Most Challenging Images
Image
Matches
Wins
Win Rate
Image
Matches
Wins
Win Rate
Fig. 7 (b)
23
23
1.00
Fig. 16 (c)
20
13
0.65
Fig. 6 (a)
22
22
1.00
Fig. 16 (b)
16
12
0.75
Fig. 12 (b)
21
21
1.00
Not shown
19
15
0.79
Fig. 16 (a)
20
20
1.00
Fig. 7 (a)
20
16
0.80
Table 8: Overall user preference win rates for our model across all baselines. The left side highlights our top performing cases where human preference was unanimously in favor of our method (100% win rate). The right side shows our most challenging edge cases, where despite struggling, our method maintained a robust minimum win rate of 65%.
Figure 10: Screenshot of our human preference study.
Acrylic
Oil
Method
FID ↓
LPIPS ↓
Clip ↑
Dinov2 ↑
DDC ↓
FID ↓
LPIPS ↓
Clip ↑
Dinov2 ↑
DDC ↓
Inverse Painting [ 9 ]
287.58
0.77
0.69
0.27
88.76
292.05
0.77
0.67
0.27
69.39
ProcessPainter [ 47 ]
223.16
0.60
0.77
0.51
38.44
257.34
0.60
0.75
0.55
21.89
PaintsUndo [ 58 ]
209.22
0.66
0.76
0.50
36.10
213.69
0.60
0.77
0.58
23.15
PaintsUndo † [ 58 ]
152.90
0.50
0.83
0.69
16.91
170.84
0.47
0.86
0.75
13.73
Ours
104.37
0.44
0.89
0.81
16.18
145.02
0.45
0.88
0.79
8.47
Table 9: Evaluation of the Acrylic and Oil painting subsets from our test set. Methods marked with † were fine-tuned on our proposed dataset. Bold and underline indicate the best and second-best results, respectively. Overall, our method consistently outperforms both zero-shot and fine-tuned baselines across metrics and art media.
Figure 11: Qualitative comparison demonstrating the effectiveness of the proposed art media transfer framework. While the base model reconstruct the input, the art media transfer model enables the application of structured artistic procedures, such as the Loomis portrait method, to real-world photographic inputs. Reference photo from CelebA-HQ [ 22 ] .
Figure 12: Nine linearly sampled frames of our method. Painting order from left to right, top to bottom. The image with the green border is the reference painting.
Figure 13: Nine linearly sampled frames of our method. Painting order from left to right, top to bottom. The image with the green border is the reference painting.
Figure 14: Comparison with ground truth (GT) of our collected dataset. Painting order from left to right, GT frame with green border is the reference frame. Image courtesy of Samir Godinjak, from the Painting with Samir YouTube channel
Figure 15: Method comparison. Painting order from left to right. The final frame (green border) is visually nearly indistinguishable from the reference image; the latter is omitted for brevity.
Figure 16: Method comparison. Painting order from left to right. The final frame (green border) is visually nearly indistinguishable from the reference image; the latter is omitted for brevity.
Figure 17: Loomis portrait drawings on CelebA-HQ [ 22 ] . The sampled keyframes illustrate the progressive, step-by-step construction of the faces using the Loomis method. The reference image is bordered in green.
Figure 18: Loomis portrait drawings on MetFaces [ 23 ] . The sampled keyframes illustrate the progressive, step-by-step construction of the faces using the Loomis method. The reference image is bordered in green.
Figure 19: Loomis portraits. The sampled keyframes illustrate the progressive, step-by-step construction of the faces using the Loomis method. The reference image, generated with Z Image Turbo [ 7 ] , bordered in green.
Figure 20: Art media transfer, from pencil to acrylic and oil. Nine linearly sampled frames of our method. Painting order from left to right, top to bottom. The image with the green border is the reference painting.
Figure 21: Pencil paintings. Nine linearly sampled frames of our method. Painting order from left to right, top to bottom. The image with the green border is the reference painting.
Digital art generation has undergone a massive paradigm shift, moving from static image synthesis to dynamic video generation that can mimic the human painting process. While existing models can synthesize visually stunning time-lapses of paintings being created or "unpainted," they largely treat the creative process as a black box. Current conditioning mechanisms are either too rigid, relying on static geometric cues like Canny edges or depth maps, or too holistic, utilizing simple scalar progress variables that offer no localized control. Consequently, users lack the agency to steer how a painting evolves over time. In this work, we argue that true control over the video generation of artwork requires a joint understanding of two fundamental dimensions: where an artist acts, and when those actions take place. To achieve this, we introduce the Painting Change Map, a novel spatio-temporal control mechanism that explicitly tracks normalized structural changes across a 1,000-timestep continuum, a resolution sufficient to capture fine-grained stroke-level changes while remaining computationally tractable. By mapping localized visual differences into an encoded RGB space, our representation enables fine-grained guidance for both the forward painting and reverse unpainting processes. Extensive benchmarking demonstrates the efficacy of our approach. We outperform existing control baselines and achieve competitive or superior results compared to state-of-the-art painting methods. Furthermore, under oracle conditions, our method surpasses all baselines by a significant margin. Code is available at https://github.com/Markus-Pobitzer/painting-change-map.
We present PaintCopilot, a co-creative neural painting assistant that models painting as an open-ended autoregressive artistic behavior conditioned on evolving canvas states and prior brushstroke history, without requiring a target image. Unlike existing neural painting methods that frame painting as pixel reconstruction toward a predefined reference, PaintCopilot predicts future strokes directly from learned artistic dynamics, analogous to how large language models continue text sequences from prior context. The framework proposes three complementary models: a ViT-based Target Predictor that infers artist intent from partial canvas observations, an autoregressive Next Stroke Predictor that generates temporally coherent brushstrokes via flow matching, and a VAE-based Region Sampler that synthesizes semantically localized stroke sequences on demand. Built on three differentiable brush representations (Hard Round, Brush Tip, and 2D Gaussian), the system supports four interactive workflows: Optimize History, Stroke Completion, Region Inpainting, and Dynamic Brush. Through case studies with professional artists, we demonstrate that PaintCopilot enables fluid co-creative painting workflows in which artists and AI continuously alternate control throughout the creative process.
Yunge Wen, Yuancheng Shen, Paul Pu Liang
MIT Media Lab, United States · New York University, United States
Unified conditional image generation requires a single model to preserve different aspects of a visual input while producing a target image that follows textual guidance. Pretrained video models provide a promising foundation by encoding visual transformations across time, yet existing approaches typically assign the condition and target to fixed temporal positions. We observe that condition-target relationships vary substantially across generation tasks, while video models naturally encode visual transformations at different temporal scales. This motivates us to model conditional generation through input-adaptive virtual temporal relations. We propose FrameForge, which captures two complementary aspects of such relations. Mixture of Temporal Experts (MoTE) models how far the target lies from the condition by dynamically selecting temporal RoPE experts with different virtual intervals. Chain of Temporal Reasoning (CoTR) models how the condition is transformed into the target through a latent intermediate state, with a reasoning-bottleneck attention mask encouraging information propagation through this pathway. Experiments on DreamBench, COCO2017, and GEdit-Bench show that FrameForge achieves superior performance in subject-driven generation, canny-to-image generation, and image editing among the compared methods.