Digital art generation has undergone a massive paradigm shift, moving from static image synthesis to dynamic video generation that can mimic the human painting process. While existing models can synthesize visually stunning time-lapses of paintings being created or "unpainted," they largely treat the creative process as a black box. Current conditioning mechanisms are either too rigid, relying on static geometric cues like Canny edges or depth maps, or too holistic, utilizing simple scalar progress variables that offer no localized control. Consequently, users lack the agency to steer how a painting evolves over time. In this work, we argue that true control over the video generation of artwork requires a joint understanding of two fundamental dimensions: where an artist acts, and when those actions take place. To achieve this, we introduce the Painting Change Map, a novel spatio-temporal control mechanism that explicitly tracks normalized structural changes across a 1,000-timestep continuum, a resolution sufficient to capture fine-grained stroke-level changes while remaining computationally tractable. By mapping localized visual differences into an encoded RGB space, our representation enables fine-grained guidance for both the forward painting and reverse unpainting processes. Extensive benchmarking demonstrates the efficacy of our approach. We outperform existing control baselines and achieve competitive or superior results compared to state-of-the-art painting methods. Furthermore, under oracle conditions, our method surpasses all baselines by a significant margin. Code is available at https://github.com/Markus-Pobitzer/painting-change-map.
Figures & tables
Figure 2: Comparison of different Painting Change Map modalities. First row: Ground truth painting process (source video from the “Painting with Samir” YouTube channel [ 25 ] ). Second row: Interpolation-based variant, where earlier painting actions map closer to white and later actions map closer to black. Third row: Pack 24-bit variant, which assigns a distinct color to each individual painting step. Fourth row: History mode, featuring independent interpolation across all three RGB channels. Fifth row: A hybrid approach, fusing the interpolation mechanism (in the R channel) with outlines extracted via a lineart processor (in the GB channels).
Global Unpainting
Local Instance Unpainting
Method
CLIP↑
DINOv2↑
FID↓
LPIPS↓
MSE↓
CLIP↑
DINOv2↑
FID↓
LPIPS↓
MSE↓
Depth-anything-v2 [ 26 ]
0.910
0.873
75.65
0.299
924.77
0.957
0.834
140.97
0.381
300.75
Canny Edges [ 4 ]
0.923
0.908
60.23
0.226
618.11
0.966
0.847
140.32
0.331
154.28
Lineart [ 6 ]
0.922
0.917
56.92
0.213
592.81
0.965
0.867
128.30
0.327
157.31
Interpolation
0.913
0.881
68.78
0.280
814.24
0.963
0.862
128.83
0.355
155.52
24Bit Packing
0.910
0.875
74.06
0.296
898.68
0.963
0.860
126.37
0.358
154.92
Table 1: Comparison of Global and Localized Control Performance. Global metrics are computed across whole unpainting, while localized control averages tasks across instance control categories (e.g., water, tree, meadow). Bold denotes overall best, underline denotes best baseline.
τ
θ
CLIP↑
DINOv2↑
LPIPS↓
MSE↓
0.05
3.0
0.925
0.916
0.215
503.51
0.05
4.0
0.925
0.916
0.214
494.96
0.13
3.0
0.925
0.920
0.208
458.90
0.13
4.0
0.926
0.920
0.207
453.25
0.13
5.0
0.926
0.920
0.207
449.68
0.18
3.0
0.925
0.919
0.209
456.42
Table 2: Ablation study on filtering thresholds. We evaluate the impact of the pixel sensitivity threshold τ (introduced in Eq. 2 ) and the global motion filtering limit θ (introduced in Eq. 3 ) using the Interpolation & Lineart modality. Bold numbers indicate the best performance.
Figure 3: Visual results of Global and Localized Control Performance. Examples related to Tab. 1 . In (a) the painting process gets reconstructed based on the change map shown in the first row and the input painting in the far right. The change map was computed from the ground truth video shown in the third row. In (b) the setup is the same but only the mountain gets unpainted keeping the background stable. This shows the generalization capabilities of the control mechanism.
Figure 4: Reconstruction of the Painting process using in the wild paintings. The input is the right most image. First the change map gets generated based on the input. In a second step the painting video is generated based on the input image and the generated change map.
Method
CLIP ↑
DINOv2 ↑
FID ↓
LPIPS ↓
MSE ↓
Params
Latency (s)
FPS
Inverse Painting [ 7 ]
0.69
0.27
287.58
0.77
10007.06
900M
333.8
0.16
ProcessPainter [ 20 ]
0.77
0.51
223.16
0.60
5253.44
1.3B
1.7
4.71
PaintsUndo [ 21 ]
0.76
0.50
209.22
0.66
9448.22
900M
129.6
0.62
Loomis Painter [ 17 ]
0.88
0.81
101.72
0.44
3152.96
14B
428.3
0.19
Interpolation & Lineart
0.87
0.74
119.53
0.45
3127.13
5B + 1.3B
130.74
0.62
Interpolation & Lineart †
0.93
0.92
55.75
0.21
449.68
1.3B
96.4
0.84
Table 3: Quantitative Comparison with State-of-the-Art Painting Generation Methods. Metrics evaluate temporal reconstruction fidelity, structural similarity, and computational efficiency. † denotes our method using ground-truth (oracle) change maps to demonstrate the performance upper bound. Bold numbers indicate the best non-oracle performance, underline the second best. Params, Latency, and FPS are only related to the generative part of each method.
Step-by-step painting tutorials are used for learning artistic techniques, but existing video resources lack interactivity and personalization. While recent generative models have advanced artistic image synthesis, they still struggle to generate temporally and structurally consistent videos. Video models excel in consistent frame generation, yet fine-tuning an image-to-video model on forward painting sequences creates a long-range dependency on the first-frame reference, disrupting its pretrained attention pattern and hindering reconstruction. We address this issue by training on reversed sequences that better preserve the reference-attention behavior. Building on this, we present Loomis Painter, a framework for reconstructing coherent painting processes. We further introduce a media-transfer training strategy that decouples input appearance from the target process, enabling generation across different artistic media. Trained on a curated dataset of real-world painting processes, our method outperforms existing baselines across LPIPS, DINO, and CLIP metrics and is consistently preferred in expert and non-expert user studies. Code available at https://github.com/Markus-Pobitzer/wlp.
Many creative multimedia systems are built upon visual primitives such as strokes or textures, which are difficult to collect at scale and fundamentally different from natural image data. This data scarcity makes it challenging for modern generative models to learn expressive and controllable primitives, limiting their use in process-aware content creation. In this work, we study the problem of learning human-like brushstroke generation from a small set of hand-drawn samples (n=470) and propose StrokeDiff, a diffusion-based framework with Smooth Regularization (SmR). SmR injects stochastic visual priors during training, providing a simple mechanism to stabilize diffusion models under sparse supervision without altering the inference process. We further show how the learned primitives can be made controllable through a Bézier-based conditioning module and integrated into a complete stroke-based painting pipeline, including prediction, generation, ordering, and compositing. This demonstrates how data-efficient primitive modeling can support expressive and structured multimedia content creation. Experiments indicate that the proposed approach produces diverse and structurally coherent brushstrokes and enables paintings with richer texture and layering, validated by both automatic metrics and human evaluation.
Dantong Qin, Alessandro Bozzon, Xian Yang +3
Delft University of Technology, Delft, Netherlands · The University of Manchester, Manchester, UK · The Hong Kong University of Science and Technology, Kowloon, HKSAR
We present PaintCopilot, a co-creative neural painting assistant that models painting as an open-ended autoregressive artistic behavior conditioned on evolving canvas states and prior brushstroke history, without requiring a target image. Unlike existing neural painting methods that frame painting as pixel reconstruction toward a predefined reference, PaintCopilot predicts future strokes directly from learned artistic dynamics, analogous to how large language models continue text sequences from prior context. The framework proposes three complementary models: a ViT-based Target Predictor that infers artist intent from partial canvas observations, an autoregressive Next Stroke Predictor that generates temporally coherent brushstrokes via flow matching, and a VAE-based Region Sampler that synthesizes semantically localized stroke sequences on demand. Built on three differentiable brush representations (Hard Round, Brush Tip, and 2D Gaussian), the system supports four interactive workflows: Optimize History, Stroke Completion, Region Inpainting, and Dynamic Brush. Through case studies with professional artists, we demonstrate that PaintCopilot enables fluid co-creative painting workflows in which artists and AI continuously alternate control throughout the creative process.
Yunge Wen, Yuancheng Shen, Paul Pu Liang
MIT Media Lab, United States · New York University, United States