cs.CVMar 1, 2026

Data-Efficient Brushstroke Generation with Diffusion Models for Oil Painting

Authors: Dantong Qin, Alessandro Bozzon, Xian Yang, Xun Zhang, Yike Guo, Pan Wang

Organizations: Delft University of Technology, Delft, Netherlands · The University of Manchester, Manchester, UK · The Hong Kong University of Science and Technology, Kowloon, HKSAR

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Many creative multimedia systems are built upon visual primitives such as strokes or textures, which are difficult to collect at scale and fundamentally different from natural image data. This data scarcity makes it challenging for modern generative models to learn expressive and controllable primitives, limiting their use in process-aware content creation. In this work, we study the problem of learning human-like brushstroke generation from a small set of hand-drawn samples (n=470) and propose StrokeDiff, a diffusion-based framework with Smooth Regularization (SmR). SmR injects stochastic visual priors during training, providing a simple mechanism to stabilize diffusion models under sparse supervision without altering the inference process. We further show how the learned primitives can be made controllable through a Bézier-based conditioning module and integrated into a complete stroke-based painting pipeline, including prediction, generation, ordering, and compositing. This demonstrates how data-efficient primitive modeling can support expressive and structured multimedia content creation. Experiments indicate that the proposed approach produces diverse and structurally coherent brushstrokes and enables paintings with richer texture and layering, validated by both automatic metrics and human evaluation.

Figures & tables

Figure 1. Comparison of stroke quality and painting outputs. (a) Prior SBR methods use geometric or template strokes with limited expressive capacity. Our strokes, learned from human-painted data, yield richer structure and texture. (b) Target oil painting style, showing abstraction beyond realism. (c) Compared to Huang et al. ( Huang et al., 2019 ) , Our method introduces a stronger artistic domain shift.
Figure 2. Overview of our method. (A) Smooth Regularization injects priors during training to mitigate mode collapse under limited data. At each timestep, a brushstroke xsx_{s} is sampled from the dataset and injected into the forward process alongside a newly sampled noise term ϵ∗∼N(0,I)\epsilon^{*}\sim\mathcal{N}(0,I) ; both terms are scaled by a factor η∼Uni[0,Υ]\eta\sim\mathrm{Uni}[0,\Upsilon] . (B) The StrokeDiff framework is trained in two stages. In Stage 1, the model learns to generate strokes with SmR but without conditional inputs. In Stage 2, we incorporate parameter conditioning via a raster-to-vector module that predicts vectorized stroke parameters (e.g., control points, opacity, RGB, width). These parameters are then used to control the generation of specific brushstroke attributes.
Figure 3. Stroke generation using Stable Diffusion v1-5 ( Rombach et al., 2022 ) fine-tuned on typical stroke data that lacks sufficient diversity. Due to severe mode collapse early in training, the model produces only texture-like images with minimal variation and no authentic brushstroke resemblance.
Figure 4
Figure 6. Examples of the collected strokes.
Table 6
Figure 7. Qualitative comparison with other approaches: 1) Neural Painters ( Nakano, 2019 ) , using VAE and GAN trained on synthetic MyPaint strokes; 2) StrokeGAN ( Wang et al., 2023 ) , a GAN trained on fourth-order Bézier curves; 3) Template strokes ( Zou et al., 2021 ; Liu et al., 2021 ; Tong et al., 2022 ; Song et al., 2024 ) , taken from a widely used set of shapes commonly adopted in SBR; and 4) RobPaint ( Bidgoli et al., 2020 ) , a VAE trained with human grayscale strokes. Our method generates clearer strokes while preserving brush textures.
Figure 8. Qualitative comparison of paintings produced by different stroke-based rendering pipelines: Diffvg ( Li et al., 2020 ) , Learn2Paint ( Huang et al., 2019 ) , Stylized ( Zou et al., 2021 ) , PT ( Liu et al., 2021 ) , and StrokeGAN ( Wang et al., 2023 ) . The figure illustrates how different stroke generation strategies affect texture appearance, stroke irregularity, and layering behavior in the final compositions. Zoom in to see differences.
Table 9
Figure 9. Intermediate canvases in the rendering process.
Figure 10. Ablation results. (a) Injection trend of stochastic and deterministic prior; (b) FID across training epochs for different Υ\Upsilon values; (c) SNR changes. We take the square root of standard SNR.
# Multiple Pairs →\rightarrow102032102032
FID148.691.654.210897.574.4
Table 5. Evaluation on different numbers of target-prior pairs for each data sample. The left and right groups correspond to cases where Υ\Upsilon is set to 0.3 and 0.5.
Figure 11. Example results from ablation experiments. Since the default prior-sample pairs are 32, all "w/o multi-pairs" settings were trained for 32 epochs to match the total number of updates, while other settings were trained for 1 epoch. Det.Det. and Sto.Sto. denote deterministic and stochastic prior injection methods. Although LoRA is not part of our ablation, we include its results in the fourth column for qualitative comparison.
Figure 14
 8% Open-Images1% Open-Images
MethodFID ↓QS ↑CLIP ↑FID ↓QS ↑CLIP ↑
Paint-by-Example11.66773.25474.12027.42127.47358.827
+ SmR ( x0x_{0} )11.46773.98173.54426.85831.14857.179
Table 6. Results on Paint-by-Example with subsets of Open-Images.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 16
Figure 15. Comparisons between our stroke generator and StrokeGAN ( Wang et al., 2023 ) , Stylized ( Zou et al., 2021 ) and RobPaint ( Bidgoli et al., 2020 ) .

Explore similar work

CardsList
Mar 29, 2026

Look, Compare and Draw: Differential Query Transformer for Automatic Oil Painting

This work introduces a new approach to automatic oil painting that emphasizes the creation of dynamic and expressive brushstrokes. A pivotal challenge lies in mitigating the duplicate and common-place strokes, which often lead to less aesthetic outcomes. Inspired by the human painting process, \ie, observing, comparing, and drawing, we incorporate differential image analysis into a neural oil painting model, allowing the model to effectively concentrate on the incremental impact of successive brushstrokes. To operationalize this concept, we propose the Differential Query Transformer (DQ-Transformer), a new architecture that leverages differentially derived image representations enriched with positional encoding to guide the stroke prediction process. This integration enables the model to maintain heightened sensitivity to local details, resulting in more refined and nuanced stroke generation. Furthermore, we incorporate adversarial training into our framework, enhancing the accuracy of stroke prediction and thereby improving the overall realism and fidelity of the synthesized paintings. Extensive qualitative evaluations, complemented by a controlled user study, validate that our DQ-Transformer surpasses existing methods in both visual realism and artistic authenticity, typically achieving these results with fewer strokes. The stroke-by-stroke painting animations are available on our project website.

Lingyu Liu, Yaxiong Wang, Li Zhu +2

School of Software, Xi’an Jiao-tong University, Xi’an, 710049, China. · School of Computer and Information Science, Hefei University of Technology, Jianghuai Advance Technology Center, Anhui Provincial Key Laboratory of Humanoid Robots. Anhui Provincial Industry Innovation Center of Humanoid Robots, Hefei, 230000, China. · Singapore Management University, 188065, Singapore +1

Topic pending
AskPDF
Apr 3, 2026

Differentiable Stroke Planning with Dual Parameterization for Efficient and High-Fidelity Painting Creation

In stroke-based rendering, search methods often get trapped in local minima due to discrete stroke placement, while differentiable optimizers lack structural awareness and produce unstructured layouts. To bridge this gap, we propose a dual representation that couples discrete polylines with continuous Bézier control points via a bidirectional mapping mechanism. This enables collaborative optimization: local gradients refine global stroke structures, while content-aware stroke proposals help escape poor local optima. Our representation further supports Gaussian-splatting-inspired initialization, enabling highly parallel stroke optimization across the image. Experiments show that our approach reduces the number of strokes by 30-50%, achieves more structurally coherent layouts, and improves reconstruction quality, while cutting optimization time by 30-40% compared to existing differentiable vectorization methods.

Jinfan Liu, Wuze Zhang, Zhangli Hu +3

Shanghai Jiao Tong University

Differentiable RenderingDifferentiable Optimization
AskPDF
May 20, 2026

PaintCopilot: Modeling Painting as Autonomous Artistic Continuation

We present PaintCopilot, a co-creative neural painting assistant that models painting as an open-ended autoregressive artistic behavior conditioned on evolving canvas states and prior brushstroke history, without requiring a target image. Unlike existing neural painting methods that frame painting as pixel reconstruction toward a predefined reference, PaintCopilot predicts future strokes directly from learned artistic dynamics, analogous to how large language models continue text sequences from prior context. The framework proposes three complementary models: a ViT-based Target Predictor that infers artist intent from partial canvas observations, an autoregressive Next Stroke Predictor that generates temporally coherent brushstrokes via flow matching, and a VAE-based Region Sampler that synthesizes semantically localized stroke sequences on demand. Built on three differentiable brush representations (Hard Round, Brush Tip, and 2D Gaussian), the system supports four interactive workflows: Optimize History, Stroke Completion, Region Inpainting, and Dynamic Brush. Through case studies with professional artists, we demonstrate that PaintCopilot enables fluid co-creative painting workflows in which artists and AI continuously alternate control throughout the creative process.

Yunge Wen, Yuancheng Shen, Paul Pu Liang

MIT Media Lab, United States · New York University, United States

Autoregressive Image GenerationHuman-AI Co-Creation
AskPDF