cs.LGMar 7, 2026

Agentic Planning with Reasoning for Image Styling via Offline RL

Authors: Subhojyoti Mukherjee, Stefano Petrangeli, Branislav Kveton, Trung Bui, Franck Dernoncourt, Arko Mukherjee

Organizations: Adobe Research, San Jose, CA, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Direct prompt-based editing often fails on complex transformations because vague and subjective prompts often require nuanced understanding of what should be changed in the image. Our core intuition is that leveraging compositional image editing tools rather than direct prompting profits from structured agent-level planning with explicit reasoning, leading to better results. This structured planning framework enables efficient offline RL post-training on quality-scored trajectories to improve performance. We present a tool-based agentic RL post-training framework that addresses this through structured planning with chain-of-thought reasoning. Our key contributions include: (1) A tool-based agentic planning methodology that combines a compositional library of orthogonal primitive transformations, structured context representation, and explicit per-step reasoning to decompose complex styling into interpretable tool sequences. (2) A synthetic data generation pipeline producing three large-scale datasets (each ∼\sim10K trajectories) with reasoning chains, plans, and quality scores, as no existing datasets provide such supervision. Our datasets and code are publicly available at the HuggingFace repository. (3) Offline RL training methods for learning planners with reasoning as our core algorithmic contributions, which consistently improve over the Edit-Only baseline in visual quality and instruction following. (4) Comprehensive evaluation across 4B and 8B parameter Qwen3-VL models showing that our methods outperform other baselines in the majority of compositional tasks, validated by human evaluations.

Figures & tables

Figure 1: Synthetic Data Generation Pipeline
Figure 2: Regular Text-4B: SW wins (78.77). Outperforms GPT-4o zero-shot baseline (grey).
Figure 3: Regular Text-8B: SW wins (77.86). Outperforms GPT-4o zero-shot baseline (grey).
Figure 4: Simple Vision-4B: RW dominates with visual grounding (79.33). Outperforms GPT-4o zero-shot baseline (grey).
Figure 5: Complex Vision-8B: DPO wins followed closely by RW and SW on diverse themes (85.41). Outperforms GPT-4o(grey).
Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

SectionPage
Visual Method Comparisons .A
9-way visual comparisons (RW/SW, DPO, RL examples) .A
Related Work .B
Appendix A: Complete Problem Formulation Details 
§A.1 Context Representation Details .C.1
§A.2 Action Space Specification .C.2
Appendix
Table 6
Figure 6: RW and SW Strengths Across Datasets: Examples where reward-aware weighting methods excel. Each row shows (left to right): Original, Baseline (B), Edit-Only (E), Standard (S), R (R), DPO (D), RW , SW , and GPT-4o Planner. Red box highlights RW and SW columns. Row 1 (Text-8B, Complex): Gallery art scene with bokeh lens effect and character depth— SW handles complex multi-element transformations. Row 2 (Vision-4B, Regular): Market scene with visual grounding— RW excels with continuous reward weighting. Row 3 (Vision-4B, Regular): Prehistoric stone age transformation— RW achieves strong temporal consistency. Row 4 (Text-4B, Regular): Temple shrine transformation— SW handles architectural and cultural elements effectively. Row 5 (Vision-8B, Complex): Canyon landscape with lens effects— SW produces superior depth and lighting. Reward-aware methods consistently outperform filtering ( R ) and preference learning ( DPO ) by leveraging continuous quality signals while preserving data diversity.
Figure 7: DPO Strengths Across Datasets: Examples where DPO outperforms other methods through fine-grained contrastive learning. Each row shows (left to right): Original, Baseline (B), Edit-Only (E), Standard (S), R ( R ), RW , SW , DPO (D), and GPT-4o Planner. Red box highlights RW and SW columns for comparison. Row 1 (Text-8B, Simple): Renaissance time period transformation from futuristic dome theater to ancient era. Row 2 (Vision-4B, Simple): Holographic future architecture combined with charcoal drawing artistic medium—a complex multi-action transformation. Row 3 (Vision-4B, Regular): Classic library with paradoxical snowstorm and summer plants visible outside windows. Row 4 (Vision-4B, Complex): Cafe transformed with surrealist art movement, dreamy textures, and soft edges. Row 5 (Text-8B, Complex): Dragon scene with subtle depth enhancement via color grading. DPO ’s contrastive learning captures fine-grained preference distinctions across diverse datasets and complexity levels, consistently producing results closer to GPT-4o reference than other methods. Edit-Only (E) shows inconsistent quality without structured action planning.
Figure 8: R Strengths Across Datasets: Examples where reward-filtering performs effectively. Each row shows (left to right): Original, Baseline (B), Edit-Only (E), Standard (S), R ( R ), RW , SW , DPO (D), and GPT-4o Planner. Red box highlights RW and SW columns for comparison. Row 1 (Text-8B, Simple): Winter to summer coastal town transformation—multi-action sequence changing location, architecture, time of day, and weather. Row 2 (Text-8B, Complex): Spatial depth enhancement with atmospheric haze and cinematic lighting. Row 3 (Text-8B, Complex): Wolf scene with winter atmosphere enhancement via cool-toned color grading and soft haze. Row 4 (Vision-8B, Regular): Urban kitchen transformed to snowy winter wonderland under blue twilight. Row 5 (Vision-4B, Simple): Bridge to Asgard Bifrost with Norse architecture and pencil sketch artistic medium. R ’s filtering strategy ( ri<3.5r_{i}<3.5 threshold) effectively removes low-quality data while maintaining training signal across diverse task types and complexity levels. Its computational efficiency and consistent quality make it attractive for large-scale training. Edit-Only (E) demonstrates the necessity of structured action planning.
Score RangeQuality TierRW Weight (at rr )DPO Usage
[4.5,5.0][4.5,5.0]Excellent2.0 (at r=5.0r{=}5.0 )Chosen
[4.0,4.5)[4.0,4.5)Good1.5 (at r=4.5r{=}4.5 )Chosen
[3.5,4.0)[3.5,4.0)Medium1.0 (at r=4.0r{=}4.0 )Neutral
[3.0,3.5)[3.0,3.5)Poor0.5 (at r=3.5r{=}3.5 )Rejected
[0,3.0)[0,3.0)Very Poor0.0Rejected
Appendix
Table 2: Reward Thresholds and Training Usage
StatisticSimple DatasetRegular Dataset
Total trajectories9,82410,142
Avg. actions per trajectory2.84.1
Avg. reward score3.923.74
% High quality ( r≥4.0r\geq 4.0 )62%48%
Action library size10 actions20 actions
Train/Val/Test split80%/10%/10%80%/10%/10%
Appendix
Table 3: Dataset Statistics
Figure 9: Simple Dataset Example - Autumn Vineyard to Spring Tulip Field. (a) Original: Autumn vineyard scene with golden foliage, wine barrels, grape clusters, and a rustic farmhouse. (b) Edited: Transformed to spring tulip field with vibrant pink tulips, modern technology, and clear morning atmosphere. The transformation successfully replaces location (vineyard→tulip field), season (autumn→spring), and era (modern→futuristic) while maintaining compositional structure.
Figure 10: Regular Dataset Example - Scandinavian Studio to Cyberpunk Industrial. (a) Original: Modern Scandinavian studio with gray sofa, wooden table, plants, and city view through large windows. (b) Edited: Transformed to cyberpunk industrial loft with neon lighting, metallic textures, and futuristic tech elements. The transformation demonstrates constraint-based planning: dramatic neon accents added while attempting to preserve natural wood tones and organic plant colors (partial success, reward 3/5).
PropertyNormal ExampleComplex Example
Number of actions33
Preservation constraints02
Overall reward4.0/5.0 (Good)3.0/5.0 (Medium)
Planning difficultyLowHigh
Execution difficultyMediumHigh
Adherence to plan5/5 (Perfect)3/5 (Partial)
Appendix
Table 4: Comparison of Normal vs. Complex Synthesis Examples
Figure 11: Complex Dataset Example - Arctic Glacier to Desert Canyon. (a) Original: Arctic glacier scene with deep crevasse revealing turquoise meltwater, surrounded by expansive snowfields and distant snow-capped mountain peaks. Strong diagonal composition with cold blue-white color palette. (b) Edited: Transformed to dramatic desert canyon with layered sandstone and rocky canyon walls in warm ochre and amber tones. The diagonal crevasse structure is preserved but now rendered as desert canyon walls. Background mountains maintain imposing presence with reddish-brown rock faces. Depth perception and spatial relationships successfully maintained through consistent atmospheric perspective (reward 5.0/5.0).
CharacteristicSimpleRegularComplex
Dataset Sizen=10,000n=10{,}000n=10,000n=10{,}000n=10,000n=10{,}000
Action Library10 actions20 actions30 actions
Action Types9 THEME + 1 STYLE10 atomic + 10 constraint30 styling & transformation
Avg Actions/Sample2-33-52-4
Theme Diversity31 locations10 interior design styles83 diverse themes
Transformation TypeSimple, 1-2 distinct changesCompositional, 3-5 interacting changesModerate, 2-4 styling changes
Appendix
Table 5: Comparison of three synthetic datasets. Normal uses atomic actions on diverse locations. Regular adds constraint/compositional actions for interior design styles. Complex expands to 30 actions and 83 diverse themes, balancing complexity with broad distribution coverage.
PropertyRWDPO
Learning signalAbsolute rewardsRelative preferences
Data usageAll dataPaired data only
OptimizationDirect likelihoodContrastive likelihood
Implicit KLThrough weightsThrough log-ratio
Sample efficiencyHighMedium
Distinction qualityCoarse-grainedFine-grained
Appendix
Table 6: Theoretical Comparison: RW vs. DPO
PropertySLRRWDPO
Uses reward signalsNoYesYesYes
Uses all dataYesNo (65%)YesPaired only
Manual tuning requiredNoneThresholdWeight fnBeta
Computational cost1 ×\times1 ×\times1 ×\times2 ×\times
Contrastive learningNoNoNoYes
Data efficiencyMediumLowHighMedium
Appendix
Table 7: Comprehensive Comparison of Training Algorithms
Figure 12: GPT-4o image quality evaluation for Regular Dataset, Text-8B models (8 methods including GPT-4o Planner). SW achieves highest Overall score among trained models (77.86), with RW winning visual quality metrics and E showing lowest performance (71.24). We outperform GPT-4o zero-shot baseline on image quality.
Figure 13: GPT-4o image quality evaluation for Simple Dataset, Vision-4B models (8 methods including GPT-4o Planner). RW achieves highest Overall score among trained models (79.33) and dominates 5/6 metrics, demonstrating strong performance with visual grounding. We outperform GPT-4o zero-shot baseline on image quality.
Figure 14: GPT-4o image quality evaluation for Simple Dataset, Vision-8B models (6 methods including GPT-4o Planner; E/ SW not evaluated). R achieves highest Overall score among trained models (79.62) and wins 4/6 metrics, with RW winning visual quality dimensions. GPT-4o zero-shot baseline achieves highest overall score on this configuration.
Figure 15: GPT-4o image quality evaluation for Regular Dataset, Text-4B models (8 methods including GPT-4o Planner). SW achieves highest Overall score among trained models on highly complex triple-action transformations, demonstrating superior handling of extreme compositional complexity. We outperform GPT-4o zero-shot baseline on image quality.
Figure 16: GPT-4o image quality evaluation for Complex Dataset, Text-8B models (8 methods including GPT-4o Planner). R achieves highest Overall score among trained models, showing that larger models enable simpler filtering strategies to handle highest complexity effectively. We outperform GPT-4o zero-shot baseline on image quality.
Figure 17: GPT-4o image quality evaluation for Regular Dataset, Vision-4B models (8 methods including GPT-4o Planner). DPO achieves highest Overall score among trained models, showing that visual grounding enables preference-based learning to excel on highest complexity. We outperform GPT-4o zero-shot baseline on image quality.
Figure 18: GPT-4o image quality evaluation for Regular Dataset, Vision-8B models (8 methods including GPT-4o Planner). DPO achieves the highest Overall score among trained models (85.41), demonstrating that preference learning reaches peak effectiveness at maximum scale with visual grounding. We outperform GPT-4o zero-shot baseline on image quality.
ConfigurationBESRRWSWD
Regular Dataset
Text-4B76.0371.4975.0377.1277.1878.7774.88
Text-8B74.7971.2474.2377.6277.3477.8675.85
Simple Dataset
Vision-4B77.2878.0477.6078.3579.3378.6578.27
Vision-8B78.07-78.7379.6278.79-78.98
Appendix
Table 8: Overall Scores Across All Configurations (7 Methods)
Figure 19: GPT-4o action plan quality evaluation for Simple Dataset, Vision-4B models: RW achieves highest Overall Score (82.09), outperforming SW (80.37), D (81.12), R (79.50), S (79.32), and B (82.01). RW dominates reasoning quality metrics: Reasoning Conciseness (88.50), Reasoning Completeness (77.14), and Reasoning Specificity (71.79), achieving Overall Reasoning Quality of 79.15. On action quality, B surprisingly leads Overall Action Quality (85.84), but RW ranks second (84.89) while excelling at reasoning. E shows N/A across all metrics, unable to generate action plans. These results demonstrate that RW ’s reward-weighting effectively improves both action planning coherence and reasoning quality on normal complexity tasks. GPT-4o Planner serves as performance ceiling with Overall Score 84.23, Overall Action Quality 87.15, and Overall Reasoning Quality 81.21.
Figure 20: GPT-4o action plan quality evaluation for Regular Dataset, Vision-4B models: SW achieves highest Overall Score (81.57), outperforming R (81.27), S (81.14), D (79.82), RW (81.10), and B (79.83). SW dominates reasoning quality: Reasoning Conciseness (88.31), Reasoning Completeness (76.92), Reasoning Specificity (72.69), achieving Overall Reasoning Quality of 79.26 (highest among all methods). SW also leads Overall Action Quality (86.39). R wins individual action metrics like Relevance (91.77) and Theme/Style Focus (78.77), while D excels at Efficiency (86.57). E shows N/A on all metrics. These results demonstrate SW ’s advantage on diverse theme distributions with broad action spaces (83 themes, 30 actions), where standardized reward weighting adapts better to compositional tasks. GPT-4o Planner serves as performance ceiling with Overall Score 83.23, Overall Action Quality 85.80, and Overall Reasoning Quality 80.57.
Figure 21: GPT-4o action plan quality evaluation for Regular Dataset, Vision-8B models: SW achieves highest Overall Score (83.08), outperforming S (82.87), R (82.58), B (82.66), RW (82.59), and D (82.57). SW dominates reasoning quality: Reasoning Conciseness (89.15), Reasoning Completeness (78.68), Reasoning Specificity (71.32), achieving Overall Reasoning Quality of 79.72 (highest). SW also leads Overall Action Quality (86.39) and wins 6/10 individual metrics including Relevance (92.25), Completeness (79.46), and Efficiency (87.60). B shows surprisingly strong performance (82.66 Overall), particularly on Reasoning Specificity (72.71). E shows N/A on all metrics. With larger models (8B) on complex datasets, SW ’s standardized weighting provides consistent advantages across both action planning and reasoning dimensions. GPT-4o Planner serves as performance ceiling with Overall Score 83.72, Overall Action Quality 86.27, and Overall Reasoning Quality 81.06.
Figure 22: Location and style transformation from desert to moss garden with pencil sketch. Left to right: Original image, Baseline result, SW result, Ground truth. Both models use 3 actions, but SW provides more detailed, contextual reasoning (shown below).
Figure 23: Complex scene transformation from European church to Angkor Wat temple with jungle and snowy winter. Left to right: Original, Baseline result, SW result, Ground truth. SW achieves the transformation with 2 actions instead of 4 by recognizing that location_setting encompasses architectural changes.
Figure 24: Contrast enhancement task for restaurant scene. Left to right: Original, Baseline result, SW result, Ground truth. Baseline refuses the task (0 actions), claiming it contradicts the photorealistic style, while SW successfully solves it with 2 actions.
ParameterValue
LoRA Rank16
LoRA Alpha32
LoRA Dropout0.05
Learning Rate2×10−52\times 10^{-5}
LR ScheduleCosine with warmup
Warmup Steps100
Appendix
Table 9: Training Hyperparameters
ConfigurationTraining TimePeak Memory
Text-Only (per method)1.5-2 hours28 GB
Vision (with caching)3-4 hours45 GB
Vision (without caching)9-12 hours45 GB
Speedup from caching3×0×
Appendix
Table 10: Training Time and Resources
Figure 25: Web-based annotation interface used by human evaluators. Annotators rate each sample on four quality dimensions (Edit Quality, Action Plan Quality, Reasoning Quality, Overall Sample Quality) using a Pass/Partial/Fail scale. The interface displays the original image (left), edited image (right), editing goal, context extraction, action plan with per-step reasoning, and synthesized instruction. Optional comments allow annotators to provide detailed feedback on edge cases.
RatingCountPercentage
Pass67277.0%
Partial13014.9%
Fail718.1%
Total873100%
Appendix
Table 11: Overall Quality Distribution Across All Samples
DatasetSamplesPassPartialFail
Regular100073.8%15.6%10.5%
Complex100077.9%13.4%8.7%
Complex100079.4%15.7%5.0%
Appendix
Table 12: Quality Distribution by Dataset Variant
Figure 26: Overall quality pass rates by annotator across three dataset variants. All annotators achieve >70%>70\% pass rates on Regular and Complex datasets, with Regular showing more variance (62.5%-87%), reflecting subjective assessment of atomic transformations. The consistent high pass rates on complex datasets validate the robustness of our synthetic data generation pipeline.
Figure 27: Distribution of agreement types across dataset variants. Exact agreement ranges from 62.8% (Regular) to 66.2% (Complex), with adjacent agreement (Pass-Partial or Partial-Fail) at 25-27%. Complete disagreement (Pass vs Fail) remains below 11% for all datasets, indicating strong consistency in fundamental quality judgments despite subjective differences on borderline cases.
Figure 28: GPT-4o validation study interface: Annotators view 7 images side-by-side (original + 6 edited versions) and select top-3 ranked methods, then rate the winner on Visual Quality, Instruction Following, and Overall Quality. Study design validates GPT-4o correlation with human judgment and identifies best-performing training methods across 279 samples.
Figure 29: Method win rates (1st place) by dataset. Left: Simple dataset shows SW leading (20.8%), followed by RW (20.8%) and D (16.7%). Center: Regular dataset shows SW winning (21.7%), followed by R (17.8%) and RW/D (16.4%). Right: Complex shows D leading (23.1%), followed by R (17.9%) and SW/S/RW ( ∼17%{\sim}17\% ). Win rate differences are small (typically 5-10 percentage points), indicating similar method performance.
MethodWinsPassPartialFail
SW8359 (71.1%)17 (20.5%)7 (8.4%)
D7759 (76.6%)11 (14.3%)7 (9.1%)
RW7458 (78.4%)11 (14.9%)5 (6.8%)
R6946 (66.7%)13 (18.8%)10 (14.5%)
S6148 (78.7%)9 (14.8%)4 (6.6%)
B4941 (83.7%)7 (14.3%)1 (2.0%)
Appendix
Table 13: Quality Distribution for Winning Samples by Method (Combined Datasets)
DatasetMean ρ\rhoWinner AccuracyTop-2 AccuracyKendall’s τ\tau
Simple0.09742.4%75.7%0.215
Regular0.12245.4%76.3%-0.043
Complex0.09053.0%82.9%0.000
Combined0.10346.9%78.3%0.057
Appendix
Table 14: GPT-4o Correlation with Human Judgment
More in this topic

Image Editing

Image editing changes selected visual content while preserving the parts of an image that should remain unchanged.

See all 315 papers

Explore similar work

CardsList

  1. From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing

    May 14, 2026Anirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae LeeText-Guided Image EditingTool-Using Vision-Language Agents

    University of Wisconsin–Madison, WI, USA · Adobe Research, San Jose, CA, USA

  2. DDA-Thinker: Decoupled Dual-Atomic Reinforcement Learning for Reasoning-Driven Image Editing

    Apr 28, 2026Hanqing Yang, Qiang Zhou, Yongchao Du +6Reinforcement LearningText-Guided Image Editing

    Alibaba Group · Zhejiang University

  3. RePlan: Reasoning-guided Region Planning for Complex Instruction-based Image Editing

    Dec 18, 2025Tianyuan Qu, Lei Ke, Xiaohang Zhan +6Text-Guided Image EditingVLM Reasoning

    Tencent AI Lab, China · The Chinese University of Hong Kong, China · The Hong Kong University of Science and Technology, China