Learning to Present: Inverse Specification Rewards for Agentic Slide Generation

Karthik Ragunath Ananda Kumar; Subrahmanyam Arunachalam

cs.AIMar 17, 2026

Learning to Present: Inverse Specification Rewards for Agentic Slide Generation

Authors: Karthik Ragunath Ananda Kumar, Subrahmanyam Arunachalam

Organizations: Tavus Inc. · University of Texas at Dallas · Texas A&M University

Abstract

Automated presentation generation remains a challenging task requiring coherent content creation, visual design, and audience-aware communication. This work proposes an OpenEnv-compatible reinforcement learning environment where LLM agents learn to research topics, plan content, and generate professional HTML slide presentations through tool use. We introduce a multi-component reward system combining structural validation, render quality assessment, LLM-based aesthetic scoring, content quality metrics, and an inverse specification reward that measures how faithfully generated slides convey their intended purpose. The inverse specification reward, an "inverse task" where an LLM attempts to recover the original specification from generated slides, provides a holistic quality signal. Our approach fine-tunes Qwen2.5-Coder-7B via GRPO, training only 0.5% of parameters on prompts derived from expert demonstrations collected using Claude Opus 4.6. Experiments on 48 diverse business briefs across six models demonstrate that our fine-tuned 7B model achieves 91.2% of Claude Opus 4.6's quality while improving 33.1% over the base model. The six-model comparison reveals that instruction adherence and tool-use compliance, rather than raw parameter count, determine agentic task performance. We contribute SlideRL, an open-source dataset of 288 multi-turn rollout trajectories across all six models: https://huggingface.co/datasets/KarthikRagunathAnandaKumar/sliderl-multi-turn-rollouts Code: https://github.com/pushing-the-frontier/slide-forge-llm

Figures & tables

Fig. 1: Architecture of the proposed system. The LLM agent working in the training loop generates tool calls that are executed in the environment, with multi-component rewards guiding policy optimization.

Category	Tools	Description
Research	web_search , fetch_url	Info gathering
Content	create_outline , revise_outline	Planning
Design	generate_slide , edit_slide , set_theme	Slide creation
Structure	get_slide_content , delete_slide , reorder_slides , duplicate_slide , insert_slide	Deck ops
Meta	review_deck , finalize	Quality control

TABLE I: Tool categories and descriptions.

Field	Type	Description
brief	SlideBrief	Task configuration
research_context	list[dict]	Accumulated research
outline	list[dict]	Slide structure
slides_html	list[str]	Generated HTML
slides_png	list[bytes]	Rendered PNGs
theme	str	Visual theme

TABLE II: Environment state representation.

Field	Type	Description
result	str	Tool execution result
success	bool	Action succeeded
current_slide_count	int	Slides created
phase	str	Workflow phase
slide_previews	list[str]	Base64 PNG thumbnails

TABLE III: Observation space fields.

Component	Weight	Description
code_rules	1.0	Structural validation
render_quality	2.0	Render success, HTML validity
aesthetic_html	1.5	HTML design quality
aesthetic_visual	1.5	Visual screenshot quality
content_quality	2.0	Topic relevance, grounding
spec_reconstruction	2.0	Inverse specification faithfulness

TABLE IV: Reward component weights and descriptions.

Fig. 2: Expert trajectory generation pipeline. The expert LLM generates a tool call each turn, which is executed in the environment. Step rewards are computed as quality deltas after each action, and the conversation history accumulates until the episode terminates.

Parameter	Value
Base model	Qwen2.5-Coder-7B-Instruct
Parameters (total)	7.61B
Quantization	4-bit NormalFloat
LoRA rank ( $r$ )	16
LoRA alpha ( $\alpha$ )	16
LoRA dropout	0.0

TABLE V: GRPO training configuration.

Fig. 3: Architecture of the base Qwen2.5-Coder-7B-Instruct model. All 7.62B parameters are frozen and stored in 4-bit quantized format. The model comprises 28 transformer decoder layers, each containing Grouped-Query Attention (28 query heads, 4 KV heads, head dim 128) and a SwiGLU feed-forward network (intermediate dim 18,944). Legend: $\ast$ frozen layers, $\ast$ trainable layers.

Fig. 4: Architecture of the GRPO-finetuned SlideRL model. LoRA adapters (rank $r{=}16$ ) are injected into all 7 linear projections per layer—Q, K, V, O (attention) and gate, up, down (FFN)—adding 1.44M trainable parameters per layer (40.4M total, 0.53% of 7.62B). Base weights remain frozen in 4-bit; only the LoRA matrices (bfloat16) are updated during GRPO training. Legend: $\ast$ frozen layers, $\ast$ trainable layers.

Model	Type	Params
Fine-tuned (Ours)	LoRA Qwen2.5-7B	7B (0.5% train.)
Base Qwen 7B	Qwen2.5-7B-Inst.	7B
Claude Opus 4.6	Proprietary	Undisclosed
Claude Sonnet 4.6	Proprietary	Undisclosed
Llama 4 Scout	Open-weight	109B (17B active)
GPT OSS 120B	Open-weight	120B

TABLE VI: Models evaluated.

Metric	Fine-tuned (Ours)	Base Qwen	Claude Opus 4.6	Claude Sonnet 4.6	Llama 4 Scout	GPT OSS 120B
Overall quality	0.724	0.544	0.794	0.775	0.779	0.249
Completion rate	46/48 (95.8%)	34/48 (70.8%)	48/48 (100%)	48/48 (100%)	48/48 (100%)	15/48 (31.2%)
Avg turns used	22.3	19.1	27.8	29.2	18.3	6.5
Avg slides created	7.0	5.2	7.3	7.3	7.3	2.2
Avg time/brief	71.6s	43.8s	393.3s	421.7s	155.4s	66.1s

TABLE VII: Aggregate results on 48 business briefs.

Fig. 5: Model ranking by overall quality.

Component	Fine-tuned	Base Qwen	Claude Opus	Claude Sonnet	Llama 4 Scout	GPT OSS 120B
code_rules	0.905	0.663	0.960	0.931	0.949	0.294
render_quality	0.958	0.708	1.000	1.000	1.000	0.309
content_quality	0.783	0.604	0.878	0.884	0.903	0.270
aesthetic_html	0.658	0.492	0.761	0.727	0.660	0.241
aesthetic_visual	0.539	0.397	0.568	0.550	0.546	0.184
spec_reconstruction	0.530	0.412	0.616	0.567	0.615	0.199

TABLE VIII: Per-component quality scores.

Fig. 7: Reward component comparison (radar chart).

Brief	Ours	Next Best	$\Delta$
Cloud Cost Optimization	0.836	Sonnet 4.6 (0.788)	$+0.048$
Content Marketing ROI	0.826	Opus 4.6 (0.824)	$+0.002$
Customer Success Metrics	0.816	Opus 4.6 (0.807)	$+0.009$
B2B Sales Automation	0.800	Opus 4.6 (0.770)	$+0.030$
Edge Computing Analysis	0.792	Base Qwen (0.781)	$+0.011$

TABLE IX: Briefs where the fine-tuned model outperforms all competitors.

Run	Dataset	Steps	Aggregate	Compl. Rate
Curated	3 traj.	100	0.623	71.2%
Curated	3 traj.	200	0.689	82.4%
Scaled	48 traj.	200	0.724	95.8%
Scaled	48 traj.	300	0.0	0%
Scaled	48 traj.	1000	0.0	0%

TABLE X: Effect of training steps and dataset scale.

Fig. 10: GRPO training loss curve for the scaled 48-trajectory run. The x-axis represents training steps; the y-axis represents the GRPO loss (clipped surrogate policy gradient loss). Consistent with the analysis above, the loss does not decrease monotonically—it oscillates within a narrow band due to the clip constraint, group-relative advantage re-centering, and online completion generation.

Steps	Avg	Min	Max	Diagnosis
0–49	$-0.953$	$-1.188$	$-0.750$	Exploration
50–99	$-0.953$	$-1.125$	$-0.750$	Still exploring
100–149	$-0.933$	$-1.188$	$-0.625$	Early learning
150–199	$-0.655$	$-1.063$	$-0.250$	Rapid improvement
200–249	$-0.593$	$-0.875$	$-0.188$	Residual diversity
250–299	$-0.375$	$-0.750$	$-0.063$	Degenerate emerging

TABLE XI: Training reward trajectory (scaled 48-trajectory run).

Fig. 11: Training reward curve for the scaled 48-trajectory GRPO run. The x-axis represents training steps; the y-axis represents the mean environment reward per step. The model exhibits consistent reward improvement from ${\approx}{-}1.0$ toward $0.0$ , demonstrating that GRPO drives meaningful policy refinement even in complex agentic settings. Early-to-mid training checkpoints (steps 100–200) capture the most behaviorally diverse and useful policies before variance narrows in later stages.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Parameters	Description
web_search	query: str	Search web
fetch_url	url: str	Fetch URL
create_outline	sections: list	Create outline
revise_outline	slide_idx, ...	Edit outline
generate_slide	slide_idx, ...	Generate slide
edit_slide	slide_idx, ...	Edit slide

Appendix

TABLE XII: Complete tool reference.

Theme	Bg	Text	Accent	Secondary
default	(255,255,255)	(33,33,33)	(41,98,255)	(100,181,246)
dark	(30,30,30)	(240,240,240)	(0,200,83)	(76,175,80)
corporate	(245,245,245)	(44,62,80)	(52,73,94)	(149,165,166)
creative	(255,253,231)	(33,33,33)	(255,87,34)	(255,167,38)
tech	(18,18,18)	(224,224,224)	(0,229,255)	(29,233,182)

Appendix

TABLE XIII: Visual theme color palettes.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 585 papers

AeSlides: Incentivizing Aesthetic Layout in LLM-Based Slide Generation via Verifiable Rewards
Apr 21, 2026Yiming Pan, Chengwei Hu, Xuancheng Huang +6LLM Alignment Reinforcement Learning with Verifiable Rewards
Beijing Institute of Technology · Zhipu AI (Z.ai)
DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation
Feb 26, 2026Hao Zheng, Guozhao Mo, Xinru Yan +7Agentic AI LLM Agents
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
SlideLab: Audience-Centered Scientific Slide Generation and Evaluation
Sep 13, 2026Vidushee Vats, Karun Sharma, Yuxia WangScientific Communication Evidence-Grounded Generation
INSAIT, Sofia University “St. Kliment Ohridski”

Apr 21, 2026J/K move · Enter open · S save

AeSlides: Incentivizing Aesthetic Layout in LLM-Based Slide Generation via Verifiable Rewards

Yiming Pan, Chengwei Hu, Xuancheng Huang, Can Huang+5

Beijing Institute of Technology · Zhipu AI (Z.ai)

LLM Alignment Reinforcement Learning with Verifiable Rewards RL for Generative Models

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated strong potential in agentic tasks, particularly in slide generation. However, slide generation poses a fundamental challenge: the generation process is text-centric, whereas its quality is governed by visual aesthetics. This modality gap leads current models to frequently produce slides with aesthetically suboptimal layouts. Existing solutions typically rely either on heavy visual reflection, which incurs high inference cost yet yields limited gains; or on fine-tuning with large-scale datasets, which still provides weak and indirect aesthetic supervision. In contrast, the explicit use of aesthetic principles as supervision remains unexplored. In this work, we present AeSlides, a reinforcement learning framework with verifiable rewards for Aesthetic layout supervision in Slide generation. We introduce a suite of meticulously designed verifiable metrics to quantify slide layout quality, capturing key layout issues in an accurate, efficient, and low-cost manner. Leveraging these verifiable metrics, we develop a GRPO-based reinforcement learning method that directly optimizes slide generation models for aesthetically coherent layouts. With only 5K training prompts on GLM-4.7-Flash, AeSlides improves aspect ratio compliance from 36% to 85%, while reducing whitespace by 44%, element collisions by 43%, and visual imbalance by 28%. Human evaluation further shows a substantial improvement in overall quality, increasing scores from 3.31 to 3.56 (+7.6%), outperforming both model-based reward optimization and reflection-based agentic approaches, and even edging out Claude-Sonnet-4.5. These results demonstrate that such a verifiable aesthetic paradigm provides an efficient and scalable approach to aligning slide generation with human aesthetic preferences. Our repository is available at https://github.com/ympan0508/aeslides.

Abstract

LLM Agent Training

Explore similar work

AeSlides: Incentivizing Aesthetic Layout in LLM-Based Slide Generation via Verifiable Rewards

DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation

SlideLab: Audience-Centered Scientific Slide Generation and Evaluation