Loomis Painter: Learning Painting Processes by Reversing Image-to-Video Generation

Markus Pobitzer; Chang Liu; Chenyi Zhuang; Teng Long; Bin Ren; Nicu Sebe

(a) Loomis-style pencil portraits painting from any portrait photo.

(b) Reconstructing the painting process while preserving the input image.

(c) Forward painting misaligns I2V attention. Our solution: reverse painting.

Figure 1: Overview of our painting process generation method. The curated video is first reversed to better align with the I2V model. A video generation model conditioned on an input image and a prompt is LoRA-tuned. The input image is a painting, and the model learns to reconstruct the steps to paint it, starting from a finished painting to a blank canvas. In (b), the media transfer model is shown, which enables to render any input image as an acrylic, oil, or pencil painting based on the text input. To achieve this, we generate variations of the reference image using image editing models and train the video generation model to reconstruct the original painting process.

Figure 2: Comparison of final frame. On the left, the forward process is not able to reconstruct the input image fully; details are still missing, i.e., bottom right part. Image courtesy Samir Godinjak, from the Painting with Samir YouTube channel.

Figure 3: Reference attention distribution (WAN I2V 5B). Probability of attending to the reference frame relative to non-self frames (

P(\text{attend to ref.}\mid\text{not self})

). Our reverse painting model maintains a decaying attention profile similar to the base model, whereas forward painting shifts focus heavily toward later frames. Fine-tuned models were trained for 7 epochs. Stepwise plateaus of 4 frames reflect the model’s

4\times

temporal latent compression. See Fig. 8 in the appendix for additional architectures.

Figure 4: Dataset Curation Pipeline Overview. Our framework extracts painting workflows from raw tutorial videos. First, start and end frames are detected, and the painting canvas is localized. The video is then partitioned into

N

segments, from which

M

frames are sampled per segment. Subsequently, occlusions (e.g., hands, brushes) are detected in the sampled frames, and a masked median is computed over the sampled frames, using the preceding median frame as a reference to remove transient obstructions. Producing

N

occlusion-free frames. Image courtesy of Samir Godinjak, from the Painting with Samir YouTube channel.

	Acrylic	Oil	Pencil	Loomis	Total
# of Videos	81	151	298	207	737
Avg. Duration [min]	$\approx$ 40	$\approx$ 30	$\approx$ 12	$\approx$ 20	—

Table 1: Overview of the curated video dataset by art medium.

Figure 5: User preference evaluation (N=22). Blue bars indicate a preference for our method, red bars indicate a preference for the baseline, and the yellow center represents a tie. Users preferred our results over ProcessPainter, Inverse Painting, and PaintsUndo in 96.2%, 78.7%, and 95.6% of comparisons, respectively.

Figure 6: Sampled frames of our method with input images. Painting order from left to right, top to bottom. The image with the green border is the reference painting. (a) Illustrates a typical painting process where the model first creates a sketch of the mountain, paints the background, layers the colors of the foreground objects, and finishes with fine details. (b) and (d) Show pencil paintings where outlines are sketched first, followed by details and depth added via hatching. (c) An acrylic painting process where the scene gets first sketched with blue color before being painted, the input image is a pencil sketch. (e) Demonstrates the Loomis portrait method, which uses simple circles and division lines to faithfully recreate the head.

Method	FID $\downarrow$	LPIPS $\downarrow$	CLIP $\uparrow$	DINOv2 $\uparrow$	DDC $\downarrow$	Latency (s)	FPS	Params
Inverse Painting [ 9 ]	326.15	0.61	0.66	0.21	48.77	333.8	0.16	900M
ProcessPainter [ 47 ]	282.90	0.53	0.76	0.50	17.79	1.7	4.71	1.3B
PaintsUndo [ 58 ]	236.52	0.55	0.77	0.56	15.05	129.6	0.62	900M
PaintsUndo † [ 58 ]	196.66	0.43	0.83	0.66	10.02	129.6	0.62	900M
Ours (VACE)	183.61	0.42	0.83	0.70	9.18	171.7	0.47	1.3B
Ours (Wan 2.2)	152.42	0.38	0.85	0.75	7.97	37.3	2.17	5B

Table 2: Evaluation using various metrics for measuring fidelity and inference speed. The number of parameters only relates to the image/video generation component. † denotes fine-tuned on our dataset. Best overall results in bold, best baseline results underlined.

Stroke Coherence
Method $\|$ Video	1	2	3	4	5	6	Avg.
PaintsUndo	$2.4$	$1.5$	$1.6$	$2.0$	$1.6$	$1.0$	$1.7_{\pm 1.1}$
Ours	$3.4$	$4.2$	$3.4$	$4.0$	$3.8$	$4.3$	$\underline{3.9_{\pm 1.0}}$
GT	$4.2$	$4.8$	$4.8$	$4.4$	$4.0$	$5.0$	$\mathbf{4.5_{\pm 0.9}}$
Layering Logic
PaintsUndo	$2.2$	$1.5$	$1.1$	$2.2$	$1.6$	$1.1$	$1.7_{\pm 1.1}$

Table 3: Expert evaluation results. Participants rated videos across five metrics on a 5-point Likert scale. Our method consistently outperforms the baseline, approaching Ground Truth (GT) in several criteria.

		Forward		Reverse (Ours)
Model	Epoch	FID $\downarrow$	LPIPS $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$
5B	1	171	0.414	187	0.436
5B	3	179	0.424	166	0.395
5B	5	175	0.424	160	0.389
5B	7	163	0.401	159	0.390
14B	7	173	0.417	164	0.393

Table 4: Ablation study evaluating the impact of our proposed reverse painting training strategy. Comparing the base video generation model against our method under the same training constraints demonstrates that the reverse painting strategy converges faster and improves visual fidelity.

Figure 7: Method comparison. Painting order from left to right. The final frame (green border) is visually nearly indistinguish- able from the reference image; the latter is omitted for brevity.

Figure 8: Reference attention distribution. Probability of attending to the reference frame relative to non-self frames (

P(\text{attend to ref.}\mid\text{not self})

). Our reverse painting model maintains a decaying attention profile similar to the base model across architectures, whereas forward painting shifts focus heavily toward later frames. Fine-tuned models were trained for 7 epochs. Stepwise plateaus of 4 frames reflect the model’s

4\times

temporal latent compression.

Method	FID $\downarrow$	LPIPS $\downarrow$	CLIP $\uparrow$	DINOv2 $\uparrow$	DDC $\downarrow$	Distance			Params
						LPIPS $\downarrow$	CLIP $\uparrow$	DINOv2 $\uparrow$
Qwen + Base	306.03	0.70	0.74	0.52	41.16	0.68	0.86	0.79	20B + 5B
Fine tuned + Base	169.32	0.45	0.84	0.73	11.97	0.24	0.96	0.97	5B + 5B
Art Media Transfer	168.78	0.44	0.84	0.73	10.36	0.22	0.97	0.97	5B
Oracle + Base	152.42	0.38	0.85	0.75	7.97	0.03	0.98	0.99	5B

Table 5: Quantitative comparison of our proposed Art Media Transfer model against baselines. Best non-oracle results are in bold . Art Media Transfer achieves the highest overall performance while requiring significantly fewer parameters. “Distance” measures visual similarity between the generated final frame and the ground truth. Oracle + Base establishes the theoretical upper bound.

Figure 9: Screenshot of the expert user study. These questions were asked for each painting video.

Model	Params	Peak VRAM (GB)	Latency (s)	FPS
Wan 2.1	14B	47.0	428.3	0.19
Wan 2.2	5B	35.3	37.3	2.17
Vace	1.3B	18.4	171.7	0.47

Table 6: Inference Efficiency Comparison. Benchmarked on a single NVIDIA A100 GPU generating 81 frames at

832\times 480

spatial resolution in bf16 precision. Best results per column are highlighted in bold .

Method A	Method B	Win (A) (%)	Tie (%)	Win (B) (%)	Total	$p$ -value
Ours	PaintsUndo	95.6	1.3	3.2	158	$<0.001$
Ours	Inverse Painting	78.7	9.8	11.6	164	$<0.001$
Ours	ProcessPainter	96.2	0.0	3.8	160	$<0.001$
PaintsUndo	Inverse Painting	27.2	14.2	58.6	162	$<0.001$
PaintsUndo	ProcessPainter	47.0	14.6	38.4	164	0.272
Inverse Painting	ProcessPainter	63.4	8.5	28.0	164	$<0.001$

Table 7: Human evaluation results for overall preference. Method A and Method B denote the two compared approaches. The

p

-values indicate statistical significance.

Top Performing Images				Most Challenging Images
Image	Matches	Wins	Win Rate	Image	Matches	Wins	Win Rate
Fig. 7 (b)	23	23	1.00	Fig. 16 (c)	20	13	0.65
Fig. 6 (a)	22	22	1.00	Fig. 16 (b)	16	12	0.75
Fig. 12 (b)	21	21	1.00	Not shown	19	15	0.79
Fig. 16 (a)	20	20	1.00	Fig. 7 (a)	20	16	0.80

Table 8: Overall user preference win rates for our model across all baselines. The left side highlights our top performing cases where human preference was unanimously in favor of our method (100% win rate). The right side shows our most challenging edge cases, where despite struggling, our method maintained a robust minimum win rate of 65%.

Figure 10: Screenshot of our human preference study.

	Acrylic					Oil
Method	FID $\downarrow$	LPIPS $\downarrow$	Clip $\uparrow$	Dinov2 $\uparrow$	DDC $\downarrow$	FID $\downarrow$	LPIPS $\downarrow$	Clip $\uparrow$	Dinov2 $\uparrow$	DDC $\downarrow$
Inverse Painting [ 9 ]	287.58	0.77	0.69	0.27	88.76	292.05	0.77	0.67	0.27	69.39
ProcessPainter [ 47 ]	223.16	0.60	0.77	0.51	38.44	257.34	0.60	0.75	0.55	21.89
PaintsUndo [ 58 ]	209.22	0.66	0.76	0.50	36.10	213.69	0.60	0.77	0.58	23.15
PaintsUndo † [ 58 ]	152.90	0.50	0.83	0.69	16.91	170.84	0.47	0.86	0.75	13.73
Ours	104.37	0.44	0.89	0.81	16.18	145.02	0.45	0.88	0.79	8.47

Table 9: Evaluation of the Acrylic and Oil painting subsets from our test set. Methods marked with † were fine-tuned on our proposed dataset. Bold and underline indicate the best and second-best results, respectively. Overall, our method consistently outperforms both zero-shot and fine-tuned baselines across metrics and art media.

Figure 11: Qualitative comparison demonstrating the effectiveness of the proposed art media transfer framework. While the base model reconstruct the input, the art media transfer model enables the application of structured artistic procedures, such as the Loomis portrait method, to real-world photographic inputs. Reference photo from CelebA-HQ [ 22 ] .

Figure 12: Nine linearly sampled frames of our method. Painting order from left to right, top to bottom. The image with the green border is the reference painting.

Figure 13: Nine linearly sampled frames of our method. Painting order from left to right, top to bottom. The image with the green border is the reference painting.

Figure 14: Comparison with ground truth (GT) of our collected dataset. Painting order from left to right, GT frame with green border is the reference frame. Image courtesy of Samir Godinjak, from the Painting with Samir YouTube channel

Figure 15: Method comparison. Painting order from left to right. The final frame (green border) is visually nearly indistinguishable from the reference image; the latter is omitted for brevity.

Figure 16: Method comparison. Painting order from left to right. The final frame (green border) is visually nearly indistinguishable from the reference image; the latter is omitted for brevity.

Figure 17: Loomis portrait drawings on CelebA-HQ [ 22 ] . The sampled keyframes illustrate the progressive, step-by-step construction of the faces using the Loomis method. The reference image is bordered in green.

Figure 18: Loomis portrait drawings on MetFaces [ 23 ] . The sampled keyframes illustrate the progressive, step-by-step construction of the faces using the Loomis method. The reference image is bordered in green.

Figure 19: Loomis portraits. The sampled keyframes illustrate the progressive, step-by-step construction of the faces using the Loomis method. The reference image, generated with Z Image Turbo [ 7 ] , bordered in green.

Figure 20: Art media transfer, from pencil to acrylic and oil. Nine linearly sampled frames of our method. Painting order from left to right, top to bottom. The image with the green border is the reference painting.

Figure 21: Pencil paintings. Nine linearly sampled frames of our method. Painting order from left to right, top to bottom. The image with the green border is the reference painting.

Loomis Painter: Learning Painting Processes by Reversing Image-to-Video Generation

Abstract

Figures & tables

Image-to-Video Generation

Painting Change Map as guidance for generating painting processes

PaintCopilot: Modeling Painting as Autonomous Artistic Continuation

FrameForge: Harnessing Priors of Video Foundation Models for Unified Image Generation

Abstract

Image-to-Video Generation

Explore similar work

Painting Change Map as guidance for generating painting processes

PaintCopilot: Modeling Painting as Autonomous Artistic Continuation

FrameForge: Harnessing Priors of Video Foundation Models for Unified Image Generation