Unlocking the Latent Canvas: Eliciting and Benchmarking Symbolic Visual Expression in LLMs

Yiren Zheng; Shibo Li; Jiaming Liu; Haofan Wang; Yiren Song

cs.CVMar 15, 2026

Unlocking the Latent Canvas: Eliciting and Benchmarking Symbolic Visual Expression in LLMs

Authors: Yiren Zheng, Shibo Li, Jiaming Liu, Haofan Wang, Yiren Song

Organizations: Tsinghua University · Fudan University · Alibaba Group · Lovart AI ·

Abstract

Current multimodal approaches predominantly treat visual generation as an external process, relying on pixel rendering or code execution, thereby overlooking the native visual representation capabilities latent within Large Language Models (LLMs). In this work, we unlock this potential through ASCII art, a compact, efficient, and text-native visual format. We introduce SVE-ASCII, a unified framework designed to elicit and benchmark Symbolic Visual Expression directly within the pure text space. To address the scarcity of systematic resources, we construct ASCIIArt-7K, a high-quality dataset synthesized via a novel "Seed-and-Evolve" pipeline that augments human-curated anchors through in-context stylistic editing. We further implement a unified instruction-tuning strategy that jointly optimizes for both Generation (Text-to-ASCII) and Understanding (ASCII-to-Text). Crucially, our experiments reveal a critical phenomenon regarding task duality: while it is established that perception aids generation, we provide compelling evidence that generative training significantly enhances visual comprehension. This confirms a mutually reinforcing cycle in symbolic visual processing, a relationship previously hypothesized but rarely empirically demonstrated in the visual domain. We release our dataset, the ASCIIArt-Bench benchmark, and the SVE-ASCII model, establishing a robust baseline for native text-based visual intelligence.

Figures & tables

Figure 2: Overview of SVE-ASCII. First, we construct the ASCIIArt-7K using a scalable synthesis pipeline. Subsequently, we fine-tune Qwen2.5-7B-Instruct via Understanding-Generation Joint Training. By switching supervision signals between tasks, our model achieves efficient bidirectional capabilities in both understanding and generating ASCII art.

Figure 3: Generation from Scratch vs. Imitation. The quality of the generated ASCII art cup improves significantly when an imitation example is provided in the prompt.

Panel of Table 1 — Table 1: Style Rules Decomposition. Taking a dog ASCII art as an example, this stage extracts the specific set of characters used and the topological features describing the dog’s shape.

Figure 4: Impact of Topology on Style Transfer. We synthesize variants for different seeds. Visually insensitive subjects (e.g., trucks) demonstrate robust style transfer across diverse shapes. Conversely, visually sensitive subjects (e.g., rabbits) struggle with large morphological changes, often failing to transform into distinct animals such as foxes or cats.

Model	Recall Subset (In-Distribution)						Generalization Subset (Out-of-Distribution)
Model	SA	IF	SC	SL	CE	Comp.	SA	IF †	SC	SL	CE	Comp.
GPT-5.2	0.558	0.544	0.785	0.690	0.825	0.633	0.604	0.568	0.756	0.713	0.801	0.650
Kimi K2	0.383	0.420	0.812	0.607	0.814	0.537	0.449	0.392	0.808	0.596	0.789	0.539
Doubao-Seed-1.6	0.259	0.265	0.694	0.450	0.838	0.413	0.332	0.347	0.698	0.476	0.845	0.465
Qwen-Plus	0.346	0.351	0.804	0.577	0.836	0.500	0.398	0.371	0.782	0.595	0.834	0.519
Gemini 3.0 Flash	0.474	0.453	0.544	0.585	0.754	0.522	0.490	0.403	0.535	0.590	0.723	0.505

Table 2: Main results on our ASCII Art Benchmark comparing Proprietary (Closed-Weight) and Open-Weight models. Consistent with the benchmark definition, we evaluate models on the Recall Subset (In-Distribution) and the Generalization Subset (Out-of-Distribution). The metrics reported are Semantic Alignment (SA), Instruction Faithfulness (IF), Structural Coherence (SC), Spatial Logic (SL), Character Efficiency (CE), and the Weighted Composite Score (Comp.). Bold indicates the best performance, and blue text indicates the second best. Inst. denotes the instruction-tuned version of the model, and Qwen2.5-7B-Inst. refers to the base model in this benchmark.

Model	Task I (Direct)			Task II (Select)
Model	Seen	Unseen	Avg.	Seen	Unseen	Avg.
GPT-5.2	11.0	17.0	14.0	18.0	17.0	17.5
Kimi-K2	9.0	6.0	7.5	10.0	5.0	7.5
Doubao-Seed-1.6	2.0	4.0	3.0	5.0	4.0	4.5
Qwen-Plus	9.0	10.0	9.5	8.0	6.0	7.0
Claude-Sonnet-4.5	16.0	14.0	15.0	19.0	21.0	20.0

Table 3: Results on the ASCII Art Understanding Benchmark comparing Proprietary (Closed-Weight) and Open-Weight models. We evaluate models on two tasks: Task I: Direct Understanding and Task II: Category Selection. All metrics are reported as Accuracy (%), evaluating performance on Seen, Unseen, and Overall. Bold indicates the best performance, and blue text indicates the second best.

Figure 5: Qualitative comparison with SOTA methods on the Generation Task. We compare the proposed method with SOTA baselines on our evaluation benchmarks, presenting results from the Recall Subset on the left and the Generalization Subset on the right.

Metric / Task	Single-Task	SVE-ASCII	$\Delta$ (%)
Part I: Generation Task (Composite Score)
Set A: Semantic Align.	0.838	0.936	$\uparrow$ 11.7%
Set A: Instr. Faith.	0.841	0.946	$\uparrow$ 12.5%
Set A: Spatial Logic	0.833	0.926	$\uparrow$ 11.2%
Set A: Composite	0.841	0.929	$\uparrow$ 10.5%
Set B: Instr. Faith. †	0.549	0.619	$\uparrow$ 12.8%

Table 4: Effect of Understanding-Generation Joint Training. We compare SVE-ASCII with models trained in a single-task setting. Set A: Recall Subset. Set B: Generalization Subset. † IF (Instruction Faithfulness) is the critical metric for the Generalization Subset, measuring robustness to novel prompts.

\Delta

denotes the relative improvement percentage.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: More results of ours and baselines.

Dec 2, 2025

ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text

Large language models (LLMs) have demonstrated several emergent behaviors with scale, including reasoning and fluency in long-form text generation. However, they continue to struggle with tasks requiring precise spatial and positional reasoning. ASCII art, a symbolic medium where characters encode structure and form, provides a unique probe of this limitation. We introduce ASCIIBench, a novel benchmark for evaluating both the generation and classification of ASCII-text images. ASCIIBench consists of a filtered dataset of 5,135 class-labeled ASCII images and is, to our knowledge, the first publicly available benchmark of its kind. Alongside the dataset, we release weights for a fine-tuned CLIP model adapted to capture ASCII structure, enabling the evaluation of LLM-generated ASCII art. Our analysis shows that cosine similarity over CLIP embeddings fails to separate most ASCII categories, yielding chance-level performance even for low-variance classes. In contrast, classes with high internal mean similarity exhibit clear discriminability, revealing that the bottleneck lies in representation rather than generational variance. These findings position ASCII art as a stress test for multimodal representations and motivate the development of new embedding methods or evaluation metrics tailored to symbolic visual modalities. All resources are available at https://github.com/ASCIIBench/ASCIIBench.

Kerry Luo, Michael Fu, Joshua Peguero +6

Algoverse AI Research

VLM Evaluation LLM Evaluation

Ask PDF

Jun 19, 2026

ASCII Art Turns LLMs into VLA Controllers

Vision--Language--Action (VLA) controllers are often built by extending vision--language models (VLMs) with action supervision, relying on multimodal backbones with large data and compute requirements. We demonstrate that a text-only large language model (LLM) can be adapted into a VLA-style controller when visual observations are rendered into a text input using an ASCII representation. This ASCII-as-vision interface enables existing training and deployment stacks for LLMs to efficiently condition on visual state, follow natural-language instructions, and produce constrained, executable actions. We fine-tune and compare multiple LLMs and VLMs across model families and scales, using both expert demonstrations from a planning-based teacher, as well as DAgger for iterative improvement. In a 2D manipulation benchmark, in both simulation and on a physical manipulator, the resulting controllers can identify task-relevant entities and plan feasible action sequences. Our results suggest that ASCII rendering can serve as a lightweight, interpretable modality bridge from images to text, complementing conventional VLA pipelines, and opening directions for VLA research with text-only backbones.

Yitao Jiang, Roy Xing, Luyang Zhao +3

Dept. of Computer Science, Dartmouth College, Hanover, NH, USA. · Dept. of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA. · Dept. of Mechanical and Aerospace Engineering, University of Houston, Houston, TX, USA.

Language Model-Based Control AI Control Robotic Manipulation Vision-Language-Action Models

Ask PDF

Aug 7, 2026

Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMs

Current evaluations do not isolate whether text-only language models can originate visual concepts before image generation. Fluent visual prose can hide visual-plan failures: an answer may appear creative while repeating familiar visual clichés or failing to specify a renderable scene. We define Visual Creative Ideation (VCI) as the ability to produce textual visual plans that are useful, expressive, and population-novel, and introduce Ekphrasis, a 400-task benchmark spanning Abstraction, Combination, Transformation, and Adaptation. Ekphrasis scores anonymized pairwise comparisons with dimension-specific checklists, aggregates preferences with Bradley-Terry models, and uses Typed Idea Graphs to convert task-specific population clichés into novelty references. Across 14 language models, VCI separates usefulness, expressiveness, and novelty rather than reducing to fluency: strong models achieve similar overall scores through different profiles, and useful plans can remain visually clichéd. A cross-modal grounding study further shows that text-level VCI ordering largely survives faithful rendering and blind image-level preference judgment, supporting Ekphrasis as a measure of visual ideation beyond prose quality.

Hongyu Luo, He Wang, Huihao Jing +6

The Hong Kong University of Science and Technology, Hong Kong SAR, China

Creativity Assessment LLM Evaluation Language Model Generation Evaluation Pairwise Preference Evaluation Text Generation

Ask PDF

Abstract

Explore similar work

ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text

ASCII Art Turns LLMs into VLA Controllers

Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMs