ChArtist: Generating Pictorial Charts with Unified Spatial and Subject Control

Shishi Xiao; Tongyu Zhou; David Laidlaw; Gromit Yeuk-Yin Chan

cs.CVMar 15, 2026

ChArtist: Generating Pictorial Charts with Unified Spatial and Subject Control

Authors: Shishi Xiao, Tongyu Zhou, David Laidlaw, Gromit Yeuk-Yin Chan

Organizations: Brown University · Adobe Research

Abstract

A pictorial chart is an effective medium for visual storytelling, seamlessly integrating visual elements with data charts. However, creating such images is challenging because the flexibility of visual elements often conflicts with the rigidity of chart structures. This process thus requires a creative deformation that maintains both data faithfulness and visual aesthetics. Current methods that extract dense structural cues from natural images (e.g., edge or depth maps) are ill-suited as conditioning signals for pictorial chart generation. We present ChArtist, a domain-specific diffusion model for generating pictorial charts automatically, offering two distinct types of control: 1) spatial control that aligns well with the chart structure, and 2) subject-driven control that respects the visual characteristics of a reference image. To achieve this, we introduce a skeleton-based spatial control representation. This representation encodes only the data-encoding information of the chart, allowing for the easy incorporation of reference visuals without a rigid outline constraint. We implement our method based on the Diffusion Transformer (DiT) and leverage an adaptive position encoding mechanism to manage these two controls. We further introduce Spatially Gated Attention to modulate the interaction between spatial control and subject control. To support the fine-tuning of pre-trained models for this task, we created a large-scale dataset of 30,000 triplets (skeleton, reference image, pictorial chart). We also propose a unified data accuracy metric to evaluate the data faithfulness of the generated charts. We believe this work demonstrates that current generative models can achieve data-driven visual storytelling by moving beyond general-purpose conditions to task-specific representations. Project page: https://chartist-ai.github.io/.

Figures & tables

Figure 2: Pipelines for ChArtist-30K Dataset Construction.

Figure 3: The spectrum of control representation based on their complexity. Top: Existing control representations for natural image; Bottom: Our proposed skeleton-based representations for pictorial chart generation, where charts (blue) are represented as skeletons (red and green lines).

Figure 4: The whole architecture of ChArtist consists of (A) a pretrained DiT-based diffusion model with two conditional-LoRAs with different positional encoding on the image inputs, where (B) one is for spatial control ( $LoRA_{S}$ ) and another for subject control ( $LoRA_{R}$ ). (C) To avoid the interference from $LoRA_{R}$ to $LoRA_{S}$ , we employ a Spatially-Gated Attention mechanism at inference time.

Figure 5: Artifacts observed when merging multiple LoRAs in parallel. $LoRA_{R}$ can distort the spatial control and disobey the chart structure (structure misalignment), and can also introduce extra visual elements beyond the spatial constraints (style leakage).

Figure 6: Illustrations of the data accuracy metric. We construct a distance field based on the chart skeleton. Then we randomly sample the points based on the ranges on the distance field, and calculate the accuracy based on a weighted F1 Score.

Figure 7: Result of spatially aligned evaluation with different control representations. (Task 1)

Method	Spatial	Subject	Method
ControlNet-Canny	3.008	2.914 (3rd)	ControlNet-Canny
ControlNet-Depth	2.969 (3rd)	2.822 (1st)	ControlNet-Depth
SDEdit	2.897 (1st)	3.085	Paint-by-Example
Inpainting	3.168	-	-
ChArtist (Ours)	2.957 (2nd)	2.845 (2nd)	ChArtist (Ours)

Table 2: Human evaluation results from controlled online study (300 participants). Average rank ranges from 1 (best) to N (worst).

Method	Bar			Line			Pie			Image Quality
Method	Data Acc $\uparrow$	DINO $\uparrow$	CLIP-I $\uparrow$	Data Acc $\uparrow$	DINO $\uparrow$	CLIP-I $\uparrow$	Data Acc $\uparrow$	DINO $\uparrow$	CLIP-I $\uparrow$	CLIP-IQA $\uparrow$	MUSIQ $\uparrow$	MAN-IQA $\uparrow$
ControlNet-Canny + IP-Adater	0.634	0.652	0.824	0.728	0.613	0.758	0.652	0.651	0.701	0.674	67.38	0.487
ControlNet-Depth + IP-Adater	0.593	0.635	0.805	0.683	0.615	0.752	0.579	0.668	0.723	0.620	54.66	0.372
Paint-by-Example	0.912	0.586	0.708	0.513	0.429	0.615	0.420	0.495	0.634	0.683	65.37	0.574
Qwen-Image-Edit	0.733	0.697	0.856	0.574	0.621	0.732	0.765	0.578	0.621	0.660	63.18	0.567
Nano Banana	0.727	0.731	0.812	0.716	0.606	0.685	0.546	0.692	0.727	0.679	65.32	0.565

Table 3: Quantitative comparison result of dual control of spatial and subject (Task 1 + Task 2).

Panel of Figure 8 — Figure 8: (a) Dual-control generation results conditioned on both spatial structure and subject reference. (b) Results of ChArtsit with different $\beta$ showcases the importance of Spatially Gated Attention of dual control in ensuring the data accuracy.

$\beta$	Data Acc $\uparrow$	DINO $\uparrow$	CLIP-T $\uparrow$
0.3	0.927	0.732	0.324
0.6	0.876	0.748	0.349
0.9	0.729	0.775	0.337

Table 4: Ablation on control factor

\beta

on line pictorial chart.

Figure 9: Comparison with current SoTA image editing models.

Panel of Figure 12 — Figure S1 : Controlled online study questions for evaluating (a) style and semantic representation quality, and (b) skeleton alignment accuracy and precision.

Method	Spatial (150 Questions)		Subject (150 Questions)		Method
Method	Avg Rank	Kendall $\tau$	Avg Rank	Kendall $\tau$	Method
ChArtist (Ours)	2.957 (2nd)	0.052	2.845 (2nd)	0.080	ChArtist (Ours)
SDEdit	2.897 (1st)	0.070	3.085	0.086	Paint-by-Example
ControlNet-Depth	2.969 (3rd)	0.058	2.822 (1st)	0.057	ControlNet-Depth
ControlNet-Canny	3.008	0.051	2.914 (3rd)	0.069	ControlNet-Canny
Inpainting	3.168	0.074	–	–	–

Table S1 : Human evaluation results from controlled online study (300 participants). Average rank ranges from 1 (best) to N (worst).

Figure S2 : Interface of ChArtist build in existing design platform. The left side is the visual library, the right side is chart and blending configuration. Canvas is in the central for user manipulation.

Figure S3 : Result of spatially aligned evaluation with different control representations. (Task 1).

Figure S4 : Result of spatially aligned evaluation with different control representations. (Task 1).

Figure S5 : Dual-control generation bar pictorial results conditioned on both spatial structure and subject reference.

Figure S6 : Dual-control generation pie pictorial results conditioned on both spatial structure and subject reference.

Figure S7 : Dual-control generation line pictorial results conditioned on both spatial structure and subject reference.

Controllable Image Generation

Controllable image generation steers images using spatial, visual, semantic, stylistic, or physical constraints beyond ordinary text prompts.

See all 215 papers

May 5, 2026

Semantic-Structural Alignment for Generative Pictorial Charts

Traditional statistical graphics are precise but often lack the visual appeal, memorability, and engagement of pictorial charts. We present a generative framework for the automated synthesis of pictorial charts that bridges the gap between semantic expression and structural faithfulness. Rather than treating charts merely as images to be stylized, we frame the problem as a dual-conditioned generation task guided by two parallel external control signals: a text prompt capturing the semantic context of the editing intent, and a context image providing the abstract statistical chart's global structure. To reinforce these controls within a Multi-Modal Diffusion Transformer, we introduce two complementary feature-level mechanisms: structural alignment to anchor spatial layouts to the input chart, and semantic alignment to transfer expressive textures from reference images. Generalizing across major visual channels (i.e., length, area, angle, and position) and diverse semantic domains, our method produces pictorial charts that are both artistically compelling and structurally consistent. Extensive quantitative evaluations and perceptual user studies demonstrate that our framework outperforms traditional controllable generation and image editing baselines, providing a foundation for high-fidelity, data-driven generative modeling in expressive visual storytelling. Project page: https://ssalign.github.io/.

Zhida Sun, Yulin Zhang, Zheng Gu +4

Visual Computing Research Center (VCC), College of Computer Science and Software Engineering (CSSE), Shenzhen University, China · VCC, CSSE, Shenzhen University, China · Shenzhen University, China +1

Data Visualization Controllable Image Generation Multimodal Diffusion Models

Ask PDF

Jul 17, 2026

ArtChart: Faithful Artistic Chart Generation with Integrated Text Rendering

Artistic charts combine data visualization with expressive marks, textures, and typography, but they are difficult for image generators: an output is useful only when its stylization preserves chart geometry, exact in-image text, and the semantic binding between labels and marks. We introduce ArtChart, a framework for faithful artistic chart generation with integrated text rendering. Given a structured chart specification and an artistic prompt, ArtChart first renders a text-free grayscale layout that encodes the target chart geometry, then trains a chart-specific control module to preserve mathematical structure. To address the remaining text and layout errors, we further refine the generation policy through GRPO-based reinforcement learning with OCR-based text rewards, VLM-based layout rewards, and aesthetic rewards. A multi-expert distillation stage reconciles these objectives by distilling single-reward experts into one balanced generation policy. We also construct ArtChart-Bench, a bilingual 2K-prompt benchmark covering four chart types, controlled value distributions, diverse label/value formats, and 15 artistic styles, together with ArtChart-Eval, a six-axis evaluation protocol measuring mathematical logic, text accuracy, text layout, aesthetics, instruction following, and readability. Experiments on ArtChart-Bench show that ArtChart consistently outperforms prompt-only, image-editing, and generic ControlNet baselines, with the largest gains on mathematical fidelity and label-layout binding while maintaining competitive visual quality. These results suggest that artistic chart generation should be evaluated as reliable visual communication rather than as generic stylized image synthesis.

Meijia Huang, Yingjie Yin, Shihao Wang +1

Ant Group

Data Visualization T2I Generation Image Generation Evaluation

Ask PDF

Aug 27, 2026

DEEPCHART: How Far are LLMs from Faithful Data-Science Chart Generation?

Faithful chart generation in real-world data-science workflows requires grounding visualizations in scattered evidence, computing chart-ready quantities, and rendering them accurately. Modern LLMs can produce visually plausible, instruction-compliant charts, yet data-level hallucinations remain difficult to detect in long, noisy, and multimodal contexts. To measure this gap, we introduce DEEPCHART, an expert-annotated benchmark of 1,482 task-conditioned chart-generation instances drawn from real-world scientific papers, financial filings, and ecosystem reports. DEEPCHART formulates chart generation as an Extract--Reason--Visualize pipeline and evaluates source-data extraction, derived-data reasoning, and chart rendering stage by stage. Experiments with state-of-the-art models show that visually plausible charts often conceal data-level hallucinations, with extraction and reasoning errors common in realistic long and multimodal settings. These findings suggest that larger context windows alone are insufficient; faithful chart generation also requires reliable evidence extraction and quantitative reasoning before rendering. Our benchmark and associated resources are available at https://github.com/tangdouer1005/DeepChart.

Jiahui tang, Kuicai Dong, Dexun Li +7

University of Science and Technology of China · Huawei Technologies Co., Ltd.

Data Visualization LLM Reliability LLM Hallucination Detection

Ask PDF

Abstract

Controllable Image Generation

Explore similar work

Semantic-Structural Alignment for Generative Pictorial Charts

ArtChart: Faithful Artistic Chart Generation with Integrated Text Rendering

DEEPCHART: How Far are LLMs from Faithful Data-Science Chart Generation?