Agentic Visual Generation: From Generative Models to Agentic Control

Yinming Huang; Shuyuan Tu; Xi Yan; Jiahao Zhan; Zihan Yang; Zhen Xing; Hui Zhang; Tiehua Zhang

cs.CVSep 6, 2026

Agentic Visual Generation: From Generative Models to Agentic Control

Authors: Yinming Huang, Shuyuan Tu, Xi Yan, Jiahao Zhan, Zihan Yang, Zhen Xing, Hui Zhang, Tiehua Zhang,

Organizations: Fudan University · Shanghai Innovative Institute · CUHK, MMLab · Wan Team, Alibaba Tongyi Lab ·

Abstract

Visual generation is evolving from generative models used through a single invocation into agentic control processes that can plan, select tools, inspect intermediate synthesized outputs, revise failures, and reuse prior experience. In most existing systems, the controller is an LLM or VLM, while visual generation models serve as tools or executors. However, existing work lacks a consistent criterion for determining when a generation system becomes agentic. Planning depth, tool use, multi-role collaboration, and reinforcement learning are often treated as evidence of agenticity, even though none of them necessarily determines which generation decisions the controller can make. We organize the field according to what the controller can directly control in the generation process. At L1 Conditioning Control, the controller prepares the input to a predetermined generator but does not control which visual operation is executed. At L2 Execution Control, it selects and invokes actual generation, editing, rendering, or other content-modifying operations. At L3 Outcome-Adaptive Control, it observes an intermediate outcome and uses that observation to change a subsequent operation within the current task. At L4 Experience-Adaptive Control, it retains experience from completed tasks and uses that experience to change decisions on future tasks. L0 Fixed Support separately denotes generators, editors, evaluators, reward models, benchmarks, and fixed pipelines without a deployed controller that makes generation-level decisions. These levels describe a progressively broader decision-making scope rather than model size, system complexity, output quality, tool or role count, or training method. Applying this framework across image, video, editing, 3D, world, slide, and user-interface generation reveals how controller capabilities have evolved and how their mechanisms are distributed across levels.

Figures & tables

\fnum@figure : Popular-paper roadmap across the L1–L4 controller levels, organized by first public release and the latest decision capability exposed by each representative system, from condition construction and operation selection to outcome-driven revision and reusable cross-task experience; the roadmap is a qualitative overview rather than a quantitative ranking.

\fnum@figure : Illustrative comparisons across adjacent controller levels. Each panel adds one decision capability to the preceding level: condition construction in L1 Conditioning Control, operation selection in L2 Execution Control, outcome-dependent revision in L3 Outcome-Adaptive Control, or cross-task experience reuse in L4 Experience-Adaptive Control. The examples are conceptual and do not constitute quantitative comparisons.

{forest}

\fnum@figure : Four-tier taxonomy and roadmap of controller decision-making scope. Part I covers L0–L2.

{forest}

\fnum@figure : Four-tier taxonomy and roadmap of controller decision-making scope (continued). Part II covers L3–L4.

Level	Scope	Test question	Controlled variables	Status
L0	Fixed Support	Is the inference path predetermined?	Generator, retriever, evaluator, benchmark	Inclusion boundary
L1	Conditioning Control	What specification reaches a fixed executor?	Prompt, layout, reference, knowledge, storyboard	Controller level
L2	Execution Control	Which visual operation is invoked?	Tool/model, mode, program, call order	Controller level
L3	Outcome-Adaptive Control	What action follows an observed result?	Repair, reroute, regenerate, rollback, stop	Controller level
L4	Experience-Adaptive Control	What changes on the next task?	Memory, skills, profiles, policy, workflow library	Controller level

TABLE I: Controller-Capability Taxonomy of Agentic Visual Generation

Behavior	Level	Decisive test
Prompt rewrite + fixed generator	L1	Scope ends at the declarative specification
Generator/editor choice	L2	Controller selects the operation that executes
Critique + repair/regeneration	L3	Outcome changes a later action in the same task
Current-task state only	$\leq$ L3	State does not cross the episode boundary
Reusable skill from a completed task	L4	Experience changes a later task’s control
RL-trained fixed generator	L0/L1	Level follows inference-time decisions, not optimization

TABLE II: Boundary Cases Under the Maximum-Causal-Reach Rule

Panel of Figure 7 — \fnum@figure : Half-year capability evolution in the structured corpus, grouped by first public release. The 2026-H2 bar contains 52 records released through August 24, 2026. Unlike a total-only trend, the stacked bars expose which kind of controller decision drives growth.

Panel of Figure 8 — \fnum@figure : Corpus cross-sections. Panel (a) is multi-label because one system may span generation and editing or multiple output types. Panel (b) assigns each system to exactly one primary controller organization. Cell shading is normalized within each row.

Panel of Figure 9 — \fnum@figure : From L0 Fixed Support to L2 Execution Control. L0 components such as ControlNet [ 43 ] execute a supplied condition along a predetermined path. L1 Conditioning Control systems such as LayoutGPT [ 15 ] construct a generator-facing specification. L2 Execution Control systems such as Visual ChatGPT [ 17 ] and ComfyUI-Copilot [ 18 ] select and invoke actual visual operations, but remain open-loop when the resulting synthesized content does not redirect the chosen route.

Category	Subcategory	Controlled decision	Representative systems
	Controlled generation and editing	Execute a supplied condition	Latent Diffusion [ 4 ] , Video Diffusion [ 7 ] , ControlNet [ 43 ] , InstructPix2Pix [ 44 ]
Generation/retrieval	Fixed retrieval	Follow a predetermined retrieval path	Re-Imagen [ 45 ]
	Data construction	Construct or filter records	AgentComp [ 46 ] , Gen-n-Val [ 47 ] , ScaleEdit-12M [ 48 ]
Data/training	Fixed generator training	Update parameters under fixed inference	DPOK [ 49 ] , AeSlides [ 50 ] , OSPO [ 51 ]
	Evaluators and reward models	Score quality or validity	ImageReward [ 52 ] , GenEval [ 21 ] , VBench [ 22 ]
Evaluation	Benchmarks	Diagnose capability or failure	AgenticVBench [ 53 ] , DECKBench [ 54 ] , 3DCodeBench [ 55 ]

TABLE III: Fine-Grained Organization of L0 Fixed Support

Category	Subcategory	Controlled decision	Representative systems
Text prompt	Prompt transformation and constraint preservation	Produce model-ready text while retaining intent and policy constraints	Promptist [ 83 ] , TIPO [ 84 ] , DiffChat [ 85 ] , POSI [ 86 ]
	Layout and region specifications	Commit boxes, regions, or spatial relations	LMD [ 14 ] , LayoutGPT [ 15 ] , LLMControl [ 87 ]
Spatial/geometric	Scene and pose specifications	Commit scene structure, assets, or poses	LLM Blueprint [ 88 ] , NaLA [ 89 ] , DAC-Pose [ 90 ]
	Retrieval policy	Decide whether, where, and what to retrieve	World-To-Image [ 16 ] , Gen-Searcher [ 91 ]
Retrievedevidence	Evidence-to-condition grounding	Associate evidence with the intended entity or attribute	Cross-modal RAG [ 92 ] , RealRAG [ 93 ] , MosAIG [ 94 ]
	Subject, action, and temporal specifications	Commit motion, identity, and action progression	Aurora [ 95 ] , TempAct [ 96 ] , AgentHOI [ 97 ]

TABLE IV: Categories and Subcategories of L1 Conditioning Control

Category	Subcategory	Controlled decision	Representative systems
	Operation and model routing	Select a generator, editor, search tool, or mode	Visual ChatGPT [ 17 ] , LLM-I [ 128 ] , SearchGen [ 129 ]
Model/tool	Multi-step workflow construction	Compose executable tool, graph, or code operations	ComfyUI-Copilot [ 18 ] , ComfyUI-R1 [ 130 ] , GenClaw [ 131 ]
	Program-level image operations	Select ordered generation or editing calls	VPGen [ 132 ] , IEAP [ 133 ] , ImageEdit-R1 [ 134 ]
Image/graphic	Element-level structured operations	Operate on layers, vectors, objects, or graphs	VisPainter [ 135 ] , MiLDEdit [ 136 ] , SceneCraft [ 137 ]
	Clip-level generation and editing operations	Select operations over a clip or media state	VideoAgent [ 138 ] , RIVER [ 139 ]
	Shot and timeline operations	Compose scene, shot, or timeline calls	The Script is All You Need [ 140 ] , DreamFactory [ 141 ] , ViMax [ 19 ]

TABLE V: Categories and Subcategories of L2 Execution Control

Category	Subcategory	Controlled decision	Representative systems
	Image and structured-content renders	Revise from visible semantic, compositional, or layout defects	RPG [ 156 ] , GenArtist [ 157 ] , PaperBanana [ 158 ]
	Video renders	Revise motion, composition, or camera from generated frames	GenMAC [ 159 ] , MotionAgent [ 160 ] , Co-Director [ 161 ]
	3D and multiview renders	Revise geometry or placement from rendered views	SceneAssistant [ 162 ] , VIGA [ 163 ] , WorldClaw [ 164 ]
Perceptualoutcome	Document and interface renders	Revise document or code from rendered appearance	PPTAgent [ 165 ] , UI2Code N [ 42 ] , VisRefiner [ 166 ]
	Program and workflow state	Revise from execution results or structured content state	ComfySearch [ 167 ] , GVR-Coder [ 168 ]
	Timeline and source state	Revise later edits from retained temporal state	ReCA [ 169 ] , LAVE [ 170 ] , Crayotter [ 171 ]

TABLE VI: Categories and Subcategories of L3 Outcome-Adaptive Control

Panel of Figure 14 — \fnum@figure : The distinction between L3 Outcome-Adaptive Control and L4 Experience-Adaptive Control. L3 systems such as GenArtist [ 157 ] and PPTAgent [ 165 ] use a generated or rendered outcome to choose a later action in the same task. L4 systems such as GenEvolve [ 27 ] and COMFYCLAW [ 28 ] transform completed-task evidence into persistent procedures or workflow skills that change control on a later independent task.

Category	Subcategory	Controlled decision	Representative systems
Capability and tool profiles	Empirical capability records	Update future model or tool routing	OctoT2I [ 26 ] , DiffusionAgent [ 350 ] , PerfGuard [ 351 ] , GenRouter [ 352 ]
Episodic and user memory	Retrieved episodes and preferences	Condition later control on prior outcomes or users	MemoGen [ 353 ] , BrandFusion [ 354 ] , UniVA [ 355 ] , MemSlides [ 356 ]
Procedures and skills	Abstracted action procedures	Reuse a distilled strategy on a later task	GenEvolve [ 27 ] , EvoDiagram [ 357 ] , SceneCraft [ 358 ]
Workflows and harnesses	Executable workflow revisions	Reuse a verified graph, program, or harness	COMFYCLAW [ 28 ] , VideoWeaver [ 359 ] , AutoDesign [ 360 ]
Policy andmodelupdates	Persistent behavioral updates	Change future control through learned parameters or policy	SIDiffAgent [ 361 ] , JarvisEvo [ 362 ] , SPIRAL [ 363 ]
	Recursive self-improvement	Change the improvement mechanism through accepted self-updates	Gödel Agent [ 364 ] , Darwin Gödel Machine [ 365 ]

TABLE VII: Categories and Subcategories of L4 Experience-Adaptive Control

\fnum@figure : Causal evaluation of agentic visual generation. The generator, available tools, budget, and evaluator remain matched while one class of controller decisions is added at each level. The comparison attributes marginal improvement to specification construction in L1 Conditioning Control, operation selection in L2 Execution Control, outcome-dependent repair in L3 Outcome-Adaptive Control, or cross-task experience reuse in L4 Experience-Adaptive Control. Content-specific benchmarks provide measurement signals but do not determine the controller level.

Level-specific question	Matched factors	Counterfactual	Evidence of added value
L0 Fixed Support: What can the fixed executor already do?	Input condition, random seeds, sampling parameters, candidate budget	Fixed executor without controller intervention	Requirement success, synthesized-content quality, human preference, cost
L1 Conditioning Control: Does specification control improve the same executor?	Generator, seeds, sampling parameters, candidate budget	Original input or fixed specification rule	Intent preservation, specification validity, generator compliance
L2 Execution Control: Does the controller select a better route?	Available models and tools, action budget, evaluator	Fixed, random, frequency-based, or oracle route	Routing regret, invalid actions, success and quality under matched cost
L3 Outcome-Adaptive Control: Does outcome feedback cause a successful repair?	Initial plan, tools, generator, evaluator, total budget	Open-loop execution without intermediate outcome access	Diagnosis precision, repair success, preservation, calibrated stopping
L4 Experience-Adaptive Control: Does completed experience improve future tasks?	Future-task distribution, tools, generator, budget	Removed, shuffled, stale, or irrelevant experience	Forward transfer, negative transfer, forgetting, retrieval and rollback success
Cross-level: What is the marginal value of added scope?	Generator, evaluator, goal set, calls and monetary cost	Add one decision class at a time from L1 to L4	Quality–cost frontier and marginal gain at each transition

TABLE VIII: Level-Conditioned Evaluation Protocol

Output type	Benchmarks / evaluators	Signals	Protocol use
Image	T2I-CompBench [ 20 ] , GenEval [ 21 ] , Draw ALL Your Imagine [ 380 ] , MME-Unify [ 381 ] , AtelierEval [ 382 ] , AgentGen-Bench [ 129 ]	Entities, counts, relations, alignment	Match generator for L1; tools, feedback, or experience for L2–L4
Video	VBench [ 22 ] , EvalCrafter [ 23 ] , AIGVE-MACS [ 383 ] , UniVA-Bench [ 355 ] , ActVideoGen-Bench [ 363 ] , CineBench [ 155 ]	Appearance, motion, temporal state, actions	Match clip budget; separate quality from repair
3D/world	SceneCraft [ 358 ] , Agentic 3D Scene Generation [ 333 ] , SPIRAL [ 363 ]	Program validity, geometry, physics, transitions	Freeze engine, assets, views, simulator
Slides	PPTBench [ 384 ] , PresentBench [ 385 ] , PPTArena [ 317 ] , PPT-Eval [ 386 ] , DynaSlide [ 316 ] , TSBench [ 312 ]	Edits, fidelity, design, editability	Freeze deck, object model, operations, render budget
Interfaces	Design2Code [ 41 ] , FronTalk [ 306 ] , Vision2Web [ 387 ]	Fidelity, runnable code, interaction, regression	Freeze browser, code environment, tests, revisions

TABLE IX: Modality-Specific Evaluators Within the Level-Conditioned Protocol

Agentic Image Generation

Agentic image generation uses reasoning, tools, search, and iterative feedback to create and refine images with greater control.

See all 43 papers

Apr 30, 2026

Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling

Recent visual generation models have made major progress in photorealism, typography, instruction following, and interactive editing, yet they still struggle with spatial reasoning, persistent state, long-horizon consistency, and causal understanding. We argue that the field should move beyond appearance synthesis toward intelligent visual generation: plausible visuals grounded in structure, dynamics, domain knowledge, and causal relations. To frame this shift, we introduce a five-level taxonomy: Atomic Generation, Conditional Generation, In-Context Generation, Agentic Generation, and World-Modeling Generation, progressing from passive renderers to interactive, agentic, world-aware generators. We analyze key technical drivers, including flow matching, unified understanding-and-generation models, improved visual representations, post-training, reward modeling, data curation, synthetic data distillation, and sampling acceleration. We further show that current evaluations often overestimate progress by emphasizing perceptual quality while missing structural, temporal, and causal failures. By combining benchmark review, in-the-wild stress tests, and expert-constrained case studies, this roadmap offers a capability-centered lens for understanding, evaluating, and advancing the next generation of intelligent visual generation systems.

Keming Wu, Zuhao Yang, Kaichen Zhang +24

1Tsinghua University · 12LMMs-Lab · 2Nanyang Technological University +9

World Models Generative Modeling Generative AI Evaluation

Ask PDF

Mar 3, 2026

VisionCreator: A Native Visual-Generation Agentic Model with Understanding, Thinking, Planning and Creation

Visual content creation tasks demand a nuanced understanding of design conventions and creative workflows-capabilities challenging for general models, while workflow-based agents lack specialized knowledge for autonomous creative planning. To overcome these challenges, we propose VisionCreator, a native visual-generation agentic model that unifies Understanding, Thinking, Planning, and Creation (UTPC) capabilities within an end-to-end learnable framework. Our work introduces four key contributions: (i) VisGenData-4k and its construction methodology using metacognition-based VisionAgent to generate high-quality creation trajectories with explicit UTPC structures; (ii) The VisionCreator agentic model, optimized through Progressive Specialization Training (PST) and Virtual Reinforcement Learning (VRL) within a high-fidelity simulated environment, enabling stable and efficient acquisition of UTPC capabilities for complex creation tasks; (iii) VisGenBench, a comprehensive benchmark featuring 1.2k test samples across diverse scenarios for standardized evaluation of multi-step visual creation capabilities; (iv) Remarkably, our VisionCreator-8B/32B models demonstrate superior performance over larger closed-source models across multiple evaluation dimensions. Overall, this work provides a foundation for future research in visual-generation agentic systems.

Jinxiang Lai, Zexin Lu, Jiajun He +11

Hong Kong University of Science and Technology · Tencent Hunyuan

Agentic Image Generation

Ask PDF

May 28, 2026

GenClaw: Code-Driven Agentic Image Generation

Image generation models have evolved from text-conditioned pixel synthesis toward multimodal agents endowed with visual comprehension and tool invocation capabilities. Yet, existing agents remain at the mercy of underlying black-box image models. Their workflow is trapped in a repetitive cycle of prompt rewriting for generation refinement, leaving them with no mechanism to directly manipulate the canvas. In essence, the potential of LLMs to serve as a genuine "brush" for precise visual construction remains largely untapped. In this paper, we propose GenClaw, a code-driven agentic image generation paradigm that empowers the agent to create like a human artist: first conceptualizing, then sketching, and finally coloring. Specifically, the agent first constructs the conceptual knowledge and context through search and reasoning. It then utilizes code (e.g., SVG, HTML, ThreeJS) to render executable visual sketches. Finally, it employs an image generation model to supplement textures, materials, and photorealism. In this workflow, code serves as a controllable intermediate canvas bridging linguistic reasoning and pixel synthesis, seamlessly integrating programmatic logic with the visual expressiveness of generative models. By transforming image generation from a black-box paradigm into a staged process akin to authentic human creation, GenClaw offers a step toward for highly controllable and interpretable visual generation systems.

Junyan Ye, Jun He, Zilong Huang +4

1Tencent Hunyuan · 2Sun Yat-sen University · 3Tsinghua University +1

Image Generation Agentic Image Generation Tool-Using Vision-Language Agents Controllable Image Generation

Ask PDF

Abstract

Agentic Image Generation

Explore similar work

Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling

VisionCreator: A Native Visual-Generation Agentic Model with Understanding, Thinking, Planning and Creation

GenClaw: Code-Driven Agentic Image Generation