AdsWorldEngine: A Self-Evolving Conversational Advertising Agent through Orchestrator and Tool Coevolution

Simiao Zuo; Chenhui Xu; Yimeng Jia; Qiang Lou; Jian Jiao; Denis Charles

cs.IRAug 13, 2026

AdsWorldEngine: A Self-Evolving Conversational Advertising Agent through Orchestrator and Tool Coevolution

Authors: Simiao Zuo, Chenhui Xu, Yimeng Jia, Qiang Lou, Jian Jiao, Denis Charles

Organizations: Microsoft Redmond, WA, USA

Abstract

Conversational advertising aims to deliver useful ads within multi-turn assistant interactions. Unlike conventional query-based advertising, where the user's intent is often expressed in a short standalone query, conversational ads must infer latent commercial intent from the current user query, the assistant response, and dialogue history while also deciding whether an ad would be helpful rather than intrusive. We propose AdsWorldEngine, an agentic framework for conversational advertising. AdsWorldEngine uses an Opportunity Gate to determine whether ads should be shown, an Orchestrator to generate commercial intents, call advertising tools, and construct a top-3 ad slate, and an Evaluator to score delivered ads for offline optimization. The central contribution is an iterative actor-tool training procedure: we first train the Orchestrator with supervised fine-tuning and agentic reinforcement learning, then use high- and low-reward rollouts to construct preference data to train tools. This creates a self-improving loop in which the system learns not only how to use advertising tools, but also how to improve them from rewarded behavior. To support subjective production decisions, we introduce label grounded judgment modeling, which trains judgment models from human labels collected under explicit guidelines. It enriches labels with thinking traces, filters inconsistent rationales through reflection, and further optimizes binary judgments with a cost sensitive GRPO variant that preserves asymmetric reward gaps. Offline, AdsWorldEngine improves diversity by 60% and relevance by 80% over the current production ad delivery system. In an online A/B test, it increases RPM by 22% and ads coverage by 74%.

Figures & tables

Figure 1. Overview of AdsWorldEngine. At serving time, the system observes the user query, dialogue history, optional user context, and the assistant response. The Opportunity Gate first decides whether ads should be shown. If the turn is eligible, the Orchestrator generates ad-seeking intents, calls advertising tools, and selects a top- $3$ slate. Offline, the Evaluator scores delivered ads and provides rewards for improving both the Orchestrator and the tools.

Field	Content
Conversation	User asks for a kid-friendly camping resort near Newton, MA with cabins, AC, kitchen, and teen activities. The assistant recommends specific campgrounds and links.
Human label	No (information-seeking conversation)
Generated trace	“… directly about commercial entities … ads for camping resorts or booking platforms would be relevant and helpful … the human label of No is incorrect …”
Reflection result	REASONING_LABEL_MISMATCH . The trace supports the opposite label and questions the given label.

Table 1. Example of data filtering. The trace is fluent but contradicts the human label, so it is removed before SFT.

Field	Content
Conversation	User: “Which shade fits me?” Bot: “Choosing the right concealer shade depends on your skin undertone, depth, and the purpose … Step 1: Identify Your Undertone … Step 2: Match Depth … Step 3: Purpose of Use … To narrow it down, could you tell me:…”
Human label	No . The user is seeking personalized shade-matching information, not a retailer or purchase option. The assistant can complete the task directly, so inserting an ad would distract from the current interaction.
GPT-5 prompt	“The user is seeking help choosing a concealer shade, which is a beauty product they would likely need to purchase . The bot is guiding them toward identifying the right shade, which could directly lead to a buying decision . Ads for concealers or beauty products would be relevant and helpful here …” Prediction: Yes (incorrect).
AdsWorldEngine (SFT only)	“The user’s message is clearly about finding the right concealer shade … the conversation is directly about selecting a product variant . The user’s goal is to identify which specific shade to choose, which is a step toward a purchase decision . An ad could support their journey rather than distract …” Prediction: Yes (incorrect).
AdsWorldEngine	“The assistant is giving general advice based on the user’s undertone, skin depth, and existing swatches. The conversation is about matching a shade to the user’s skin, not about finding where to buy concealer . The focus is on completing a task that the assistant can handle directly , so inserting ads would distract from the user’s intent.” Prediction: No (correct).

Table 2. Example of Opportunity Gate. The prompted GPT-5 judge and AdsWorldEngine (SFT only) overreact to product language and produce a false positive. AdsWorldEngine recognizes informational intent and correctly suppresses the trigger.

Model	$\Delta$ TPR ( $\uparrow$ )	$\Delta$ FPR ( $\downarrow$ )	$\Delta$ Balanced Acc. ( $\uparrow$ )
GPT-5 prompt	–	–	–
AdsWorldEngine (SFT)	$-17.19\%$	$\mathbf{-75.42\%}$	$-4.09\%$
AdsWorldEngine	$\mathbf{0.00\%}$	$-39.07\%$	$\mathbf{+2.51\%}$

Table 3. Relative changes in True Positive Rate (TPR), False Positive Rate (FPR), and Balanced Accuracy.

Model	$\Delta$ TPR ( $\uparrow$ )	$\Delta$ FPR ( $\downarrow$ )	$\Delta$ Balanced Acc. ( $\uparrow$ )
GPT-5 prompt	–	–	–
AdsWorldEngine (SFT)	$+10.04\%$	$+6.36\%$	$+3.39\%$
AdsWorldEngine	$-0.78\%$	$\mathbf{-12.71\%}$	$\mathbf{+7.51\%}$

Table 4. Relative changes in True Positive Rate (TPR), False Positive Rate (FPR), and Balanced Accuracy.

Round	System state	Diversity	Relevance
–	Production System	–	–
$0$	AdsWorldEngine (SFT)	$+15.31\%$	$+49.37\%$
$1$	Orchestrator update	$+29.24\%$	$+53.15\%$
$1$	Tool update	$+53.90\%$	$+70.07\%$
$2$	Orchestrator update	$+58.74\%$	$+75.60\%$
$2$	Tool update	$+59.47\%$	$+78.04\%$

Table 5. Relative diversity and relevance gains over the Production System of AdsWorldEngine (30B-A3B).

Conversation

Previous user: Compare the Honda Pilot vs the Toyota Grand Hylander both 2026 Previous bot: Here is a clean, decision ready comparison … I can also break this down by family use , towing , … Current user: Yes please do Current bot: Absolutely—let’s take this comparison deeper and make it genuinely useful for a real world decision …

AdsWorldEngine (SFT) Conversation to ads relevance: lower

<think> … I think the cleanest final intent is that the user is researching and comparing the 2026 Honda Pilot and 2026 Toyota Grand Highlander to help decide which is better for their needs … the strongest queries are ones that directly target side by side comparisons and reviews … </think> <tool_call> 2026 Honda Pilot vs Toyota Grand Highlander comparison, 2026 Honda Pilot vs Toyota Grand Highlander family review </tool_call>

AdsWorldEngine Conversation to ads relevance: higher

<think> I think the cleanest final intent is that the user wants detailed, scenario based information to help decide between the 2026 Honda Pilot and the 2026 Toyota Grand Highlander … The only hard constraints are the two named vehicles and model year; everything else is open for exploration … For ads queries, the strongest grounded targets are the two specific SUV models for 2026. Each is a distinct product family, so having one query for each covers the full scope without redundancy … </think> <tool_call> 2026 Honda Pilot, 2026 Toyota Grand Highlander </tool_call>

Table 6. Case study comparing generated intents before and after reinforcement learning. After the generated intents are used to call the tools, the reflection step selects three ads, which the Evaluator scores for relevance. The model trained with SFT and GRPO achieves a stronger relevance outcome than the SFT-only model.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	$\Delta$ vs. Triggering Model
DROP EM	+2.09 pp
HaluEval Accuracy	+1.95 pp
HellaSwag	+2.64 pp
MMLU	+1.46 pp
MuSR	+4.36 pp
TruthfulQA	+5.02 pp

Appendix

Table 7. General-domain changes from the Triggering Model baseline after OPD.

Benchmark	$\Delta$ vs. Relevance Model
DROP EM	+4.06 pp
HaluEval Accuracy	+4.40 pp
HellaSwag	+4.25 pp
MMLU	+1.51 pp
MuSR	+0.53 pp
TruthfulQA	+0.98 pp

Appendix

Table 8. General-domain changes from the Relevance Model baseline after OPD.

Model	$\Delta$ TPR ( $\uparrow$ )	$\Delta$ FPR ( $\downarrow$ )	$\Delta$ ACC ( $\uparrow$ )
Triggering Model	–	–	–
Triggering + General OPD	$0.00$ pp	$-0.11$ pp	$+0.10$ pp

Appendix

Table 9. Triggering changes after general-domain OPD, using the Triggering Model as the baseline.

Model	$\Delta$ TPR ( $\uparrow$ )	$\Delta$ TNR ( $\uparrow$ )	$\Delta$ ACC ( $\uparrow$ )
Relevance Model	–	–	–
Relevance + General OPD	$+1.28$ pp	$-0.40$ pp	$+0.93$ pp

Appendix

Table 10. Relevance changes after general-domain OPD, using the Relevance Model as the baseline.

Explore similar work

Interactor: Agentic RL oriented Iterative Creation for Ad Description Generation in Sponsored Search
Jun 14, 2026Penghui Wei, Jiayu Wu, Chao Ye +3Online Advertising RL for Generative Models
Search Ads, Baidu Inc.
Evaluating and Pricing Advertisements in AI-Generated Responses
Jul 30, 2026John L. Turner-Smith, Zimeng Huang, Yuhan Fu +2Mechanism Design Online Advertising
Carnegie Mellon University, Pittsburgh, PA, USA · College of AI, Tsinghua University, Beijing, China
LERA: LLM-Enhanced RAG for Ad Auction in Generative Chatbots
May 15, 2026Haoran Sun, Xinrui Song, Xinyu Zhang +7Retrieval-Augmented Generation Mechanism Design
Peking University · Shandong University · Alibaba Group

Jun 14, 2026J/K move · Enter open · S save

Interactor: Agentic RL oriented Iterative Creation for Ad Description Generation in Sponsored Search

Penghui Wei, Jiayu Wu, Chao Ye, Zhi Guo+2

Search Ads, Baidu Inc.

Online Advertising RL for Generative Models Text Generation

Open paper AskPDF arXiv

This paper focuses on automatically generating informative ad descriptions in sponsored search. Unlike ad titles which are usually optimized to attract user click feedbacks, ad descriptions have a longer text span and possess the potential of incorporating world knowledge to address user search intents while presenting the fine-grained selling points of the ads. We propose Interactor, a multi-turn iterative creation framework optimized with agentic RL for ad description generation. The generation model acts as a policy that interacts with a customized environment consisting of multiple generative reward models. Given initial generations by the policy, the customized GenRMs evaluate multi-dimensional qualities including knowledge capacity and landing page consistency, providing both binary signals and reasoning feedbacks. The policy then iteratively refines the descriptions based on such feedbacks to ensure continuous improvement. Experiments on industrial datasets show that the Interactor framework significantly outperforms state-of-the-art approaches in generating knowledge-rich and faithful ad descriptions. Since May 2026, it has been deployed online in a leading search ads system, contributing to both ad revenue and user experience.

Abstract

Explore similar work

Interactor: Agentic RL oriented Iterative Creation for Ad Description Generation in Sponsored Search

Evaluating and Pricing Advertisements in AI-Generated Responses

LERA: LLM-Enhanced RAG for Ad Auction in Generative Chatbots