Organizations: School of Software, Xi’an Jiao-tong University, Xi’an, 710049, China. · School of Computer and Information Science, Hefei University of Technology, Jianghuai Advance Technology Center, Anhui Provincial Key Laboratory of Humanoid Robots. Anhui Provincial Industry Innovation Center of Humanoid Robots, Hefei, 230000, China. · Singapore Management University, 188065, Singapore · Faculty of Science and Technology, and Institute of Collaborative Innovation, University of Macau, Macau, 999078, China.
This work introduces a new approach to automatic oil painting that emphasizes the creation of dynamic and expressive brushstrokes. A pivotal challenge lies in mitigating the duplicate and common-place strokes, which often lead to less aesthetic outcomes. Inspired by the human painting process, \ie, observing, comparing, and drawing, we incorporate differential image analysis into a neural oil painting model, allowing the model to effectively concentrate on the incremental impact of successive brushstrokes. To operationalize this concept, we propose the Differential Query Transformer (DQ-Transformer), a new architecture that leverages differentially derived image representations enriched with positional encoding to guide the stroke prediction process. This integration enables the model to maintain heightened sensitivity to local details, resulting in more refined and nuanced stroke generation. Furthermore, we incorporate adversarial training into our framework, enhancing the accuracy of stroke prediction and thereby improving the overall realism and fidelity of the synthesized paintings. Extensive qualitative evaluations, complemented by a controlled user study, validate that our DQ-Transformer surpasses existing methods in both visual realism and artistic authenticity, typically achieving these results with fewer strokes. The stroke-by-stroke painting animations are available on our project website.
Figures & tables
Fig. 1: Differential image-guided inference process. We present four intermediate stages of oil painting according to a real target image (left). Each stage is illustrated with a diagram, where the top-left corner shows the current canvas, the top-right corner displays the corresponding differential image for that stage, and the bottom part presents the painting result inferred by our model. We observe that since we explicitly compare the content in the differential images during training, our model tends to add strokes in areas where discrepancies are more pronounced, thereby progressively reducing the discrepancy content within the differential images.
Fig. 2: A brief overview of our painter framework. Given the canvas image Ic and the target image It generated by the renderer, we first obtain their differential image Id by simply subtracting one input from the other. Three local encoders comprised of convolutional neural networks are employed to extract image features Fc,Ft, and Fd with positional information. DQ-Transformer has two components, i . e ., the DQ-encoder and the DQ-decoder. These visual features Fc,Ft and Fd , are concatenated and then fed to the DQ-encoder to obtain the fused feature Fkv . Next, we transform the differential image features Fd into query tokens to query the key and value pairs generated by the fused feature Fkv . Finally, the DQ-Transformer outputs a set of predicted strokes St^ , each accompanied by its respective confidence Ct^ . The predicted image It^ is generated by rendering these strokes onto the canvas. The discriminator operates by treating the target images It as real samples and the predicted images It^ as fake samples.
Fig. 3: Our painting progress following a coarse-to-fine manner.
Fig. 4: Qualitative comparison between our model and state-of-the-art neural painting methods on unseen real-world datasets at different levels of stroke counts. The actual number of strokes used in the painting is annotated in the top right corner of the image. Our method leverages the difference image as a dynamic query for each painting step. This observation-first approach enables our model to achieve superior visual quality with relatively fewer strokes, effectively reproducing complex details with high fidelity. Please zoom in to obtain a more detailed view.
Landscape
FFHQ
Wiki Art
Average
Stroke
Method
Lpixel↓
Lpcpt↓
Lpixel↓
Lpcpt↓
Lpixel↓
Lpcpt↓
Lpixel↓
Lpcpt↓
Stylized Neural Painting
0.068
0.941
0.057
1.047
0.064
0.998
0.063
0.995
Paint Transformer
0.080
0.851
0.067
1.052
0.072
0.934
0.073
0.946
500
Im2Oil
0.096
0.992
0.077
1.071
0.089
1.036
0.087
1.033
Learning To Paint
0.065
0.793
0.050
0.850
0.062
0.833
0.059
0.825
Compositional Neural Painter
0.069
0.886
0.053
0.996
0.062
0.907
0.062
0.930
TABLE I: Quantitative comparison with competitive methods under pixel-level and perception-level reconstruction on unseen real-world datasets at different levels of stroke counts. Lower values indicate better reconstruction. Bold indicates best. Benefiting from the observation-first mechanism, our model adapts to varying stroke budgets while preserving fine-grained details and global structure, consistently achieving strong performance across a wide range of stroke counts. This shows its robustness and efficiency in high-fidelity neural painting under resource constraints.
Fig. 5: Ablation study on the primary components of our framework at different stroke counts. The actual number of brushstrokes used in the painting is annotated in the top right corner of the image. Please zoom in to obtain a more detailed view.
Fig. 6: The average user voting rate and inference time for each method. Methods positioned closer to the upper left corner are characterized by higher user votes and faster inference speeds. Our approach surpasses the comparison methods in preference score by a clear margin and also offers faster inference speed.
Landscape
FFHQ
Wiki Art
Average
Stroke
Method
Lpixel↓
Lpcpt↓
Lpixel↓
Lpcpt↓
Lpixel↓
Lpcpt↓
Lpixel↓
Lpcpt↓
w/o Id
0.095
0.883
0.088
1.079
0.090
0.935
0.091
0.966
w/o Reg ( λc=0 )
0.085
0.849
0.077
1.048
0.081
0.907
0.081
0.935
500
w/o CoordConv
0.103
0.935
0.104
1.145
0.101
0.994
0.103
1.025
w/o Discriminator
0.070
0.920
0.062
1.037
0.065
0.975
0.066
0.977
Ours (Full)
0.063
0.751
0.051
0.881
0.058
0.812
0.057
0.815
TABLE II: Quantitative Effect of Primary Components at different levels of stroke counts. w/o Id denotes that we do not use the differential image, while w/o Reg ( λc=0 ) means the model without confidence regularization in Eq. 5 , w/o CoordConv represents we solely employ conventional convolutional layers to extract image features, w/o Discriminator denotes that we train the model without the discriminator.
Method
SNP
PT
Im2Oil
L2P
CNP
Ours
Training ( hours )
11
4
-
50
90
10
Inference ( seconds )
89
0.70
125
3
12
0.72
TABLE III: Comparison of training and inference time across different painting methods.
λc
0.05
0.1
0.2
0.5
1
5
10
Lpixel↓
0.048
0.046
0.046
0.050
0.050
0.055
0.058
Lpcpt↓
0.668
0.607
0.614
0.686
0.685
0.786
0.791
TABLE IV: Ablation study on the weight λc . We set λc=0.1 as the default value.
Fig. 7: Attention visualization comparing three configurations. (1) Paint Transformer uses fixed learnable queries and key-value from the target and current canvas. (2) A variant with static queries and key-value from target, canvas, and differential image. (3) Our DQ-Transformer uses the differential image as dynamic query and combines all three inputs for key-value. Our model generates attention maps that sharply focus on regions with significant reconstruction errors.
Fig. 8: Failure case under an extremely limited stroke budget. With only two strokes, the model produces a highly abstract output that cannot capture the structure of the target image, illustrating a fundamental limitation of stroke-based neural painting methods.
Many creative multimedia systems are built upon visual primitives such as strokes or textures, which are difficult to collect at scale and fundamentally different from natural image data. This data scarcity makes it challenging for modern generative models to learn expressive and controllable primitives, limiting their use in process-aware content creation. In this work, we study the problem of learning human-like brushstroke generation from a small set of hand-drawn samples (n=470) and propose StrokeDiff, a diffusion-based framework with Smooth Regularization (SmR). SmR injects stochastic visual priors during training, providing a simple mechanism to stabilize diffusion models under sparse supervision without altering the inference process. We further show how the learned primitives can be made controllable through a Bézier-based conditioning module and integrated into a complete stroke-based painting pipeline, including prediction, generation, ordering, and compositing. This demonstrates how data-efficient primitive modeling can support expressive and structured multimedia content creation. Experiments indicate that the proposed approach produces diverse and structurally coherent brushstrokes and enables paintings with richer texture and layering, validated by both automatic metrics and human evaluation.
Dantong Qin, Alessandro Bozzon, Xian Yang +3
Delft University of Technology, Delft, Netherlands · The University of Manchester, Manchester, UK · The Hong Kong University of Science and Technology, Kowloon, HKSAR
We present PaintCopilot, a co-creative neural painting assistant that models painting as an open-ended autoregressive artistic behavior conditioned on evolving canvas states and prior brushstroke history, without requiring a target image. Unlike existing neural painting methods that frame painting as pixel reconstruction toward a predefined reference, PaintCopilot predicts future strokes directly from learned artistic dynamics, analogous to how large language models continue text sequences from prior context. The framework proposes three complementary models: a ViT-based Target Predictor that infers artist intent from partial canvas observations, an autoregressive Next Stroke Predictor that generates temporally coherent brushstrokes via flow matching, and a VAE-based Region Sampler that synthesizes semantically localized stroke sequences on demand. Built on three differentiable brush representations (Hard Round, Brush Tip, and 2D Gaussian), the system supports four interactive workflows: Optimize History, Stroke Completion, Region Inpainting, and Dynamic Brush. Through case studies with professional artists, we demonstrate that PaintCopilot enables fluid co-creative painting workflows in which artists and AI continuously alternate control throughout the creative process.
Yunge Wen, Yuancheng Shen, Paul Pu Liang
MIT Media Lab, United States · New York University, United States
In stroke-based rendering, search methods often get trapped in local minima due to discrete stroke placement, while differentiable optimizers lack structural awareness and produce unstructured layouts. To bridge this gap, we propose a dual representation that couples discrete polylines with continuous Bézier control points via a bidirectional mapping mechanism. This enables collaborative optimization: local gradients refine global stroke structures, while content-aware stroke proposals help escape poor local optima. Our representation further supports Gaussian-splatting-inspired initialization, enabling highly parallel stroke optimization across the image. Experiments show that our approach reduces the number of strokes by 30-50%, achieves more structurally coherent layouts, and improves reconstruction quality, while cutting optimization time by 30-40% compared to existing differentiable vectorization methods.