cs.CVSep 6, 2026
SaveAgentic Visual Generation: From Generative Models to Agentic Control
Organizations: Fudan University · Shanghai Innovative Institute · CUHK, MMLab · Wan Team, Alibaba Tongyi Lab ·
Abstract
Visual generation is evolving from generative models used through a single invocation into agentic control processes that can plan, select tools, inspect intermediate synthesized outputs, revise failures, and reuse prior experience. In most existing systems, the controller is an LLM or VLM, while visual generation models serve as tools or executors. However, existing work lacks a consistent criterion for determining when a generation system becomes agentic. Planning depth, tool use, multi-role collaboration, and reinforcement learning are often treated as evidence of agenticity, even though none of them necessarily determines which generation decisions the controller can make. We organize the field according to what the controller can directly control in the generation process. At L1 Conditioning Control, the controller prepares the input to a predetermined generator but does not control which visual operation is executed. At L2 Execution Control, it selects and invokes actual generation, editing, rendering, or other content-modifying operations. At L3 Outcome-Adaptive Control, it observes an intermediate outcome and uses that observation to change a subsequent operation within the current task. At L4 Experience-Adaptive Control, it retains experience from completed tasks and uses that experience to change decisions on future tasks. L0 Fixed Support separately denotes generators, editors, evaluators, reward models, benchmarks, and fixed pipelines without a deployed controller that makes generation-level decisions. These levels describe a progressively broader decision-making scope rather than model size, system complexity, output quality, tool or role count, or training method. Applying this framework across image, video, editing, 3D, world, slide, and user-interface generation reveals how controller capabilities have evolved and how their mechanisms are distributed across levels.
Figures & tables
\fnum@figure : Popular-paper roadmap across the L1–L4 controller levels, organized by first public release and the latest decision capability exposed by each representative system, from condition construction and operation selection to outcome-driven revision and reusable cross-task experience; the roadmap is a qualitative overview rather than a quantitative ranking.
\fnum@figure : Illustrative comparisons across adjacent controller levels. Each panel adds one decision capability to the preceding level: condition construction in L1 Conditioning Control, operation selection in L2 Execution Control, outcome-dependent revision in L3 Outcome-Adaptive Control, or cross-task experience reuse in L4 Experience-Adaptive Control. The examples are conceptual and do not constitute quantitative comparisons.
| {forest} |
| {forest} |
\fnum@figure : Four-tier taxonomy and roadmap of controller decision-making scope. Part I covers L0–L2.
| {forest} |
| {forest} |
\fnum@figure : Four-tier taxonomy and roadmap of controller decision-making scope (continued). Part II covers L3–L4.
| Level | Scope | Test question | Controlled variables | Status |
|---|---|---|---|---|
| L0 | Fixed Support | Is the inference path predetermined? | Generator, retriever, evaluator, benchmark | Inclusion boundary |
| L1 | Conditioning Control | What specification reaches a fixed executor? | Prompt, layout, reference, knowledge, storyboard | Controller level |
| L2 | Execution Control | Which visual operation is invoked? | Tool/model, mode, program, call order | Controller level |
| L3 | Outcome-Adaptive Control | What action follows an observed result? | Repair, reroute, regenerate, rollback, stop | Controller level |
| L4 | Experience-Adaptive Control | What changes on the next task? | Memory, skills, profiles, policy, workflow library | Controller level |
TABLE I: Controller-Capability Taxonomy of Agentic Visual Generation
| Behavior | Level | Decisive test |
|---|---|---|
| Prompt rewrite + fixed generator | L1 | Scope ends at the declarative specification |
| Generator/editor choice | L2 | Controller selects the operation that executes |
| Critique + repair/regeneration | L3 | Outcome changes a later action in the same task |
| Current-task state only | L3 | State does not cross the episode boundary |
| Reusable skill from a completed task | L4 | Experience changes a later task’s control |
| RL-trained fixed generator | L0/L1 | Level follows inference-time decisions, not optimization |
TABLE II: Boundary Cases Under the Maximum-Causal-Reach Rule
\fnum@figure : Half-year capability evolution in the structured corpus, grouped by first public release. The 2026-H2 bar contains 52 records released through August 24, 2026. Unlike a total-only trend, the stacked bars expose which kind of controller decision drives growth.
\fnum@figure : Corpus cross-sections. Panel (a) is multi-label because one system may span generation and editing or multiple output types. Panel (b) assigns each system to exactly one primary controller organization. Cell shading is normalized within each row.
\fnum@figure : From L0 Fixed Support to L2 Execution Control. L0 components such as ControlNet [ 43 ] execute a supplied condition along a predetermined path. L1 Conditioning Control systems such as LayoutGPT [ 15 ] construct a generator-facing specification. L2 Execution Control systems such as Visual ChatGPT [ 17 ] and ComfyUI-Copilot [ 18 ] select and invoke actual visual operations, but remain open-loop when the resulting synthesized content does not redirect the chosen route.
| Category | Subcategory | Controlled decision | Representative systems |
|---|---|---|---|
| Controlled generation and editing | Execute a supplied condition | Latent Diffusion [ 4 ] , Video Diffusion [ 7 ] , ControlNet [ 43 ] , InstructPix2Pix [ 44 ] | |
| Generation/retrieval | Fixed retrieval | Follow a predetermined retrieval path | Re-Imagen [ 45 ] |
| Data construction | Construct or filter records | AgentComp [ 46 ] , Gen-n-Val [ 47 ] , ScaleEdit-12M [ 48 ] | |
| Data/training | Fixed generator training | Update parameters under fixed inference | DPOK [ 49 ] , AeSlides [ 50 ] , OSPO [ 51 ] |
| Evaluators and reward models | Score quality or validity | ImageReward [ 52 ] , GenEval [ 21 ] , VBench [ 22 ] | |
| Evaluation | Benchmarks | Diagnose capability or failure | AgenticVBench [ 53 ] , DECKBench [ 54 ] , 3DCodeBench [ 55 ] |
TABLE III: Fine-Grained Organization of L0 Fixed Support
| Category | Subcategory | Controlled decision | Representative systems |
| Text prompt | Prompt transformation and constraint preservation | Produce model-ready text while retaining intent and policy constraints | Promptist [ 83 ] , TIPO [ 84 ] , DiffChat [ 85 ] , POSI [ 86 ] |
| Layout and region specifications | Commit boxes, regions, or spatial relations | LMD [ 14 ] , LayoutGPT [ 15 ] , LLMControl [ 87 ] | |
| Spatial/geometric | Scene and pose specifications | Commit scene structure, assets, or poses | LLM Blueprint [ 88 ] , NaLA [ 89 ] , DAC-Pose [ 90 ] |
| Retrieval policy | Decide whether, where, and what to retrieve | World-To-Image [ 16 ] , Gen-Searcher [ 91 ] | |
| Retrievedevidence | Evidence-to-condition grounding | Associate evidence with the intended entity or attribute | Cross-modal RAG [ 92 ] , RealRAG [ 93 ] , MosAIG [ 94 ] |
| Subject, action, and temporal specifications | Commit motion, identity, and action progression | Aurora [ 95 ] , TempAct [ 96 ] , AgentHOI [ 97 ] |
TABLE IV: Categories and Subcategories of L1 Conditioning Control
| Category | Subcategory | Controlled decision | Representative systems |
| Operation and model routing | Select a generator, editor, search tool, or mode | Visual ChatGPT [ 17 ] , LLM-I [ 128 ] , SearchGen [ 129 ] | |
| Model/tool | Multi-step workflow construction | Compose executable tool, graph, or code operations | ComfyUI-Copilot [ 18 ] , ComfyUI-R1 [ 130 ] , GenClaw [ 131 ] |
| Program-level image operations | Select ordered generation or editing calls | VPGen [ 132 ] , IEAP [ 133 ] , ImageEdit-R1 [ 134 ] | |
| Image/graphic | Element-level structured operations | Operate on layers, vectors, objects, or graphs | VisPainter [ 135 ] , MiLDEdit [ 136 ] , SceneCraft [ 137 ] |
| Clip-level generation and editing operations | Select operations over a clip or media state | VideoAgent [ 138 ] , RIVER [ 139 ] | |
| Shot and timeline operations | Compose scene, shot, or timeline calls | The Script is All You Need [ 140 ] , DreamFactory [ 141 ] , ViMax [ 19 ] |
TABLE V: Categories and Subcategories of L2 Execution Control
| Category | Subcategory | Controlled decision | Representative systems |
| Image and structured-content renders | Revise from visible semantic, compositional, or layout defects | RPG [ 156 ] , GenArtist [ 157 ] , PaperBanana [ 158 ] | |
| Video renders | Revise motion, composition, or camera from generated frames | GenMAC [ 159 ] , MotionAgent [ 160 ] , Co-Director [ 161 ] | |
| 3D and multiview renders | Revise geometry or placement from rendered views | SceneAssistant [ 162 ] , VIGA [ 163 ] , WorldClaw [ 164 ] | |
| Perceptualoutcome | Document and interface renders | Revise document or code from rendered appearance | PPTAgent [ 165 ] , UI2Code N [ 42 ] , VisRefiner [ 166 ] |
| Program and workflow state | Revise from execution results or structured content state | ComfySearch [ 167 ] , GVR-Coder [ 168 ] | |
| Timeline and source state | Revise later edits from retained temporal state | ReCA [ 169 ] , LAVE [ 170 ] , Crayotter [ 171 ] |
TABLE VI: Categories and Subcategories of L3 Outcome-Adaptive Control
\fnum@figure : The distinction between L3 Outcome-Adaptive Control and L4 Experience-Adaptive Control. L3 systems such as GenArtist [ 157 ] and PPTAgent [ 165 ] use a generated or rendered outcome to choose a later action in the same task. L4 systems such as GenEvolve [ 27 ] and COMFYCLAW [ 28 ] transform completed-task evidence into persistent procedures or workflow skills that change control on a later independent task.
| Category | Subcategory | Controlled decision | Representative systems |
| Capability and tool profiles | Empirical capability records | Update future model or tool routing | OctoT2I [ 26 ] , DiffusionAgent [ 350 ] , PerfGuard [ 351 ] , GenRouter [ 352 ] |
| Episodic and user memory | Retrieved episodes and preferences | Condition later control on prior outcomes or users | MemoGen [ 353 ] , BrandFusion [ 354 ] , UniVA [ 355 ] , MemSlides [ 356 ] |
| Procedures and skills | Abstracted action procedures | Reuse a distilled strategy on a later task | GenEvolve [ 27 ] , EvoDiagram [ 357 ] , SceneCraft [ 358 ] |
| Workflows and harnesses | Executable workflow revisions | Reuse a verified graph, program, or harness | COMFYCLAW [ 28 ] , VideoWeaver [ 359 ] , AutoDesign [ 360 ] |
| Policy andmodelupdates | Persistent behavioral updates | Change future control through learned parameters or policy | SIDiffAgent [ 361 ] , JarvisEvo [ 362 ] , SPIRAL [ 363 ] |
| Recursive self-improvement | Change the improvement mechanism through accepted self-updates | Gödel Agent [ 364 ] , Darwin Gödel Machine [ 365 ] |
TABLE VII: Categories and Subcategories of L4 Experience-Adaptive Control
\fnum@figure : Causal evaluation of agentic visual generation. The generator, available tools, budget, and evaluator remain matched while one class of controller decisions is added at each level. The comparison attributes marginal improvement to specification construction in L1 Conditioning Control, operation selection in L2 Execution Control, outcome-dependent repair in L3 Outcome-Adaptive Control, or cross-task experience reuse in L4 Experience-Adaptive Control. Content-specific benchmarks provide measurement signals but do not determine the controller level.
| Level-specific question | Matched factors | Counterfactual | Evidence of added value |
|---|---|---|---|
| L0 Fixed Support: What can the fixed executor already do? | Input condition, random seeds, sampling parameters, candidate budget | Fixed executor without controller intervention | Requirement success, synthesized-content quality, human preference, cost |
| L1 Conditioning Control: Does specification control improve the same executor? | Generator, seeds, sampling parameters, candidate budget | Original input or fixed specification rule | Intent preservation, specification validity, generator compliance |
| L2 Execution Control: Does the controller select a better route? | Available models and tools, action budget, evaluator | Fixed, random, frequency-based, or oracle route | Routing regret, invalid actions, success and quality under matched cost |
| L3 Outcome-Adaptive Control: Does outcome feedback cause a successful repair? | Initial plan, tools, generator, evaluator, total budget | Open-loop execution without intermediate outcome access | Diagnosis precision, repair success, preservation, calibrated stopping |
| L4 Experience-Adaptive Control: Does completed experience improve future tasks? | Future-task distribution, tools, generator, budget | Removed, shuffled, stale, or irrelevant experience | Forward transfer, negative transfer, forgetting, retrieval and rollback success |
| Cross-level: What is the marginal value of added scope? | Generator, evaluator, goal set, calls and monetary cost | Add one decision class at a time from L1 to L4 | Quality–cost frontier and marginal gain at each transition |
TABLE VIII: Level-Conditioned Evaluation Protocol
| Output type | Benchmarks / evaluators | Signals | Protocol use |
|---|---|---|---|
| Image | T2I-CompBench [ 20 ] , GenEval [ 21 ] , Draw ALL Your Imagine [ 380 ] , MME-Unify [ 381 ] , AtelierEval [ 382 ] , AgentGen-Bench [ 129 ] | Entities, counts, relations, alignment | Match generator for L1; tools, feedback, or experience for L2–L4 |
| Video | VBench [ 22 ] , EvalCrafter [ 23 ] , AIGVE-MACS [ 383 ] , UniVA-Bench [ 355 ] , ActVideoGen-Bench [ 363 ] , CineBench [ 155 ] | Appearance, motion, temporal state, actions | Match clip budget; separate quality from repair |
| 3D/world | SceneCraft [ 358 ] , Agentic 3D Scene Generation [ 333 ] , SPIRAL [ 363 ] | Program validity, geometry, physics, transitions | Freeze engine, assets, views, simulator |
| Slides | PPTBench [ 384 ] , PresentBench [ 385 ] , PPTArena [ 317 ] , PPT-Eval [ 386 ] , DynaSlide [ 316 ] , TSBench [ 312 ] | Edits, fidelity, design, editability | Freeze deck, object model, operations, render budget |
| Interfaces | Design2Code [ 41 ] , FronTalk [ 306 ] , Vision2Web [ 387 ] | Fidelity, runnable code, interaction, regression | Freeze browser, code environment, tests, revisions |
TABLE IX: Modality-Specific Evaluators Within the Level-Conditioned Protocol
More in this topic
Agentic Image Generation
Agentic image generation uses reasoning, tools, search, and iterative feedback to create and refine images with greater control.
See all 43 papers