AdsWorldEngine: A Self-Evolving Conversational Advertising Agent through Orchestrator and Tool Coevolution

Simiao Zuo; Chenhui Xu; Yimeng Jia; Qiang Lou; Jian Jiao; Denis Charles

cs.IRAug 13, 2026

AdsWorldEngine: A Self-Evolving Conversational Advertising Agent through Orchestrator and Tool Coevolution

Authors: Simiao Zuo, Chenhui Xu, Yimeng Jia, Qiang Lou, Jian Jiao, Denis Charles

Organizations: Microsoft Redmond, WA, USA

Abstract

Conversational advertising aims to deliver useful ads within multi-turn assistant interactions. Unlike conventional query-based advertising, where the user's intent is often expressed in a short standalone query, conversational ads must infer latent commercial intent from the current user query, the assistant response, and dialogue history while also deciding whether an ad would be helpful rather than intrusive. We propose AdsWorldEngine, an agentic framework for conversational advertising. AdsWorldEngine uses an Opportunity Gate to determine whether ads should be shown, an Orchestrator to generate commercial intents, call advertising tools, and construct a top-3 ad slate, and an Evaluator to score delivered ads for offline optimization. The central contribution is an iterative actor-tool training procedure: we first train the Orchestrator with supervised fine-tuning and agentic reinforcement learning, then use high- and low-reward rollouts to construct preference data to train tools. This creates a self-improving loop in which the system learns not only how to use advertising tools, but also how to improve them from rewarded behavior. To support subjective production decisions, we introduce label grounded judgment modeling, which trains judgment models from human labels collected under explicit guidelines. It enriches labels with thinking traces, filters inconsistent rationales through reflection, and further optimizes binary judgments with a cost sensitive GRPO variant that preserves asymmetric reward gaps. Offline, AdsWorldEngine improves diversity by 60% and relevance by 80% over the current production ad delivery system. In an online A/B test, it increases RPM by 22% and ads coverage by 74%.

Figures & tables

Figure 1. Overview of AdsWorldEngine. At serving time, the system observes the user query, dialogue history, optional user context, and the assistant response. The Opportunity Gate first decides whether ads should be shown. If the turn is eligible, the Orchestrator generates ad-seeking intents, calls advertising tools, and selects a top- $3$ slate. Offline, the Evaluator scores delivered ads and provides rewards for improving both the Orchestrator and the tools.

Field	Content
Conversation	User asks for a kid-friendly camping resort near Newton, MA with cabins, AC, kitchen, and teen activities. The assistant recommends specific campgrounds and links.
Human label	No (information-seeking conversation)
Generated trace	“… directly about commercial entities … ads for camping resorts or booking platforms would be relevant and helpful … the human label of No is incorrect …”
Reflection result	REASONING_LABEL_MISMATCH . The trace supports the opposite label and questions the given label.

Table 1. Example of data filtering. The trace is fluent but contradicts the human label, so it is removed before SFT.

Field	Content
Conversation	User: “Which shade fits me?” Bot: “Choosing the right concealer shade depends on your skin undertone, depth, and the purpose … Step 1: Identify Your Undertone … Step 2: Match Depth … Step 3: Purpose of Use … To narrow it down, could you tell me:…”
Human label	No . The user is seeking personalized shade-matching information, not a retailer or purchase option. The assistant can complete the task directly, so inserting an ad would distract from the current interaction.
GPT-5 prompt	“The user is seeking help choosing a concealer shade, which is a beauty product they would likely need to purchase . The bot is guiding them toward identifying the right shade, which could directly lead to a buying decision . Ads for concealers or beauty products would be relevant and helpful here …” Prediction: Yes (incorrect).
AdsWorldEngine (SFT only)	“The user’s message is clearly about finding the right concealer shade … the conversation is directly about selecting a product variant . The user’s goal is to identify which specific shade to choose, which is a step toward a purchase decision . An ad could support their journey rather than distract …” Prediction: Yes (incorrect).
AdsWorldEngine	“The assistant is giving general advice based on the user’s undertone, skin depth, and existing swatches. The conversation is about matching a shade to the user’s skin, not about finding where to buy concealer . The focus is on completing a task that the assistant can handle directly , so inserting ads would distract from the user’s intent.” Prediction: No (correct).

Table 2. Example of Opportunity Gate. The prompted GPT-5 judge and AdsWorldEngine (SFT only) overreact to product language and produce a false positive. AdsWorldEngine recognizes informational intent and correctly suppresses the trigger.

Model	$\Delta$ TPR ( $\uparrow$ )	$\Delta$ FPR ( $\downarrow$ )	$\Delta$ Balanced Acc. ( $\uparrow$ )
GPT-5 prompt	–	–	–
AdsWorldEngine (SFT)	$-17.19\%$	$\mathbf{-75.42\%}$	$-4.09\%$
AdsWorldEngine	$\mathbf{0.00\%}$	$-39.07\%$	$\mathbf{+2.51\%}$

Table 3. Relative changes in True Positive Rate (TPR), False Positive Rate (FPR), and Balanced Accuracy.

Model	$\Delta$ TPR ( $\uparrow$ )	$\Delta$ FPR ( $\downarrow$ )	$\Delta$ Balanced Acc. ( $\uparrow$ )
GPT-5 prompt	–	–	–
AdsWorldEngine (SFT)	$+10.04\%$	$+6.36\%$	$+3.39\%$
AdsWorldEngine	$-0.78\%$	$\mathbf{-12.71\%}$	$\mathbf{+7.51\%}$

Table 4. Relative changes in True Positive Rate (TPR), False Positive Rate (FPR), and Balanced Accuracy.

Round	System state	Diversity	Relevance
–	Production System	–	–
$0$	AdsWorldEngine (SFT)	$+15.31\%$	$+49.37\%$
$1$	Orchestrator update	$+29.24\%$	$+53.15\%$
$1$	Tool update	$+53.90\%$	$+70.07\%$
$2$	Orchestrator update	$+58.74\%$	$+75.60\%$
$2$	Tool update	$+59.47\%$	$+78.04\%$

Table 5. Relative diversity and relevance gains over the Production System of AdsWorldEngine (30B-A3B).

Conversation

Previous user: Compare the Honda Pilot vs the Toyota Grand Hylander both 2026 Previous bot: Here is a clean, decision ready comparison … I can also break this down by family use , towing , … Current user: Yes please do Current bot: Absolutely—let’s take this comparison deeper and make it genuinely useful for a real world decision …

AdsWorldEngine (SFT) Conversation to ads relevance: lower

<think> … I think the cleanest final intent is that the user is researching and comparing the 2026 Honda Pilot and 2026 Toyota Grand Highlander to help decide which is better for their needs … the strongest queries are ones that directly target side by side comparisons and reviews … </think> <tool_call> 2026 Honda Pilot vs Toyota Grand Highlander comparison, 2026 Honda Pilot vs Toyota Grand Highlander family review </tool_call>

AdsWorldEngine Conversation to ads relevance: higher

<think> I think the cleanest final intent is that the user wants detailed, scenario based information to help decide between the 2026 Honda Pilot and the 2026 Toyota Grand Highlander … The only hard constraints are the two named vehicles and model year; everything else is open for exploration … For ads queries, the strongest grounded targets are the two specific SUV models for 2026. Each is a distinct product family, so having one query for each covers the full scope without redundancy … </think> <tool_call> 2026 Honda Pilot, 2026 Toyota Grand Highlander </tool_call>

Table 6. Case study comparing generated intents before and after reinforcement learning. After the generated intents are used to call the tools, the reflection step selects three ads, which the Evaluator scores for relevance. The model trained with SFT and GRPO achieves a stronger relevance outcome than the SFT-only model.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	$\Delta$ vs. Triggering Model
DROP EM	+2.09 pp
HaluEval Accuracy	+1.95 pp
HellaSwag	+2.64 pp
MMLU	+1.46 pp
MuSR	+4.36 pp
TruthfulQA	+5.02 pp

Appendix

Table 7. General-domain changes from the Triggering Model baseline after OPD.

Benchmark	$\Delta$ vs. Relevance Model
DROP EM	+4.06 pp
HaluEval Accuracy	+4.40 pp
HellaSwag	+4.25 pp
MMLU	+1.51 pp
MuSR	+0.53 pp
TruthfulQA	+0.98 pp

Appendix

Table 8. General-domain changes from the Relevance Model baseline after OPD.

Model	$\Delta$ TPR ( $\uparrow$ )	$\Delta$ FPR ( $\downarrow$ )	$\Delta$ ACC ( $\uparrow$ )
Triggering Model	–	–	–
Triggering + General OPD	$0.00$ pp	$-0.11$ pp	$+0.10$ pp

Appendix

Table 9. Triggering changes after general-domain OPD, using the Triggering Model as the baseline.

Model	$\Delta$ TPR ( $\uparrow$ )	$\Delta$ TNR ( $\uparrow$ )	$\Delta$ ACC ( $\uparrow$ )
Relevance Model	–	–	–
Relevance + General OPD	$+1.28$ pp	$-0.40$ pp	$+0.93$ pp

Appendix

Table 10. Relevance changes after general-domain OPD, using the Relevance Model as the baseline.

Explore similar work

Jun 14, 2026cs.CL

Interactor: Agentic RL oriented Iterative Creation for Ad Description Generation in Sponsored Search

This paper focuses on automatically generating informative ad descriptions in sponsored search. Unlike ad titles which are usually optimized to attract user click feedbacks, ad descriptions have a longer text span and possess the potential of incorporating world knowledge to address user search intents while presenting the fine-grained selling points of the ads. We propose Interactor, a multi-turn iterative creation framework optimized with agentic RL for ad description generation. The generation model acts as a policy that interacts with a customized environment consisting of multiple generative reward models. Given initial generations by the policy, the customized GenRMs evaluate multi-dimensional qualities including knowledge capacity and landing page consistency, providing both binary signals and reasoning feedbacks. The policy then iteratively refines the descriptions based on such feedbacks to ensure continuous improvement. Experiments on industrial datasets show that the Interactor framework significantly outperforms state-of-the-art approaches in generating knowledge-rich and faithful ad descriptions. Since May 2026, it has been deployed online in a leading search ads system, contributing to both ad revenue and user experience.

Penghui Wei, Jiayu Wu, Chao Ye +3

Search Ads, Baidu Inc.

Online Advertising RL for Generative Models Text Generation

Ask PDF

Jul 30, 2026cs.AI

Evaluating and Pricing Advertisements in AI-Generated Responses

As search increasingly shifts toward LLM-driven answer engines, advertising is becoming embedded within the generated response itself and should therefore be evaluated for both user utility and commercial value. The key challenge is click-through intent: behavioural logs are unavailable, human annotation resists calibration, and frontier LLM judges conflate intent with linguistic fluency. These gaps compound, as principled pricing presupposes a continuous intent signal, while generating such a signal presupposes supervision that is currently unavailable. We construct the missing supervision through a psychologically grounded agent simulation framework, and distil it into a parameter-efficient evaluator that predicts click-through intent, together with the three companion dimensions of ad quality, as smooth, differentiable estimates. Validated through sign-certain behavioural perturbations, the evaluator surpasses frontier zero-shot judges on relevance sensitivity (79% versus 60-67%), tracks graded content degradation, generalises without error to 103 fictional products, and agrees with human preference in 86% of pairwise judgements across five annotators, with agreement rising in the evaluator's confidence. Upon its estimates we build the pricing layer directly, deriving the unique payment rule under which truthful bidding is optimal, demonstrating it on a best-of-k allocation, and extending the mechanism to non-monotone allocations. The same differentiable signal stands ready as a training objective for ad generation.

John L. Turner-Smith, Zimeng Huang, Yuhan Fu +2

Carnegie Mellon University, Pittsburgh, PA, USA · College of AI, Tsinghua University, Beijing, China

Mechanism Design Online Advertising Intent Classification CTR Prediction Automated Evaluation Generative AI Evaluation

Ask PDF

May 15, 2026cs.IR

LERA: LLM-Enhanced RAG for Ad Auction in Generative Chatbots

The integration of advertising auction mechanisms into large language model (LLM)-based chatbots presents a significant opportunity for commercialization, yet poses unique challenges in balancing relevance, efficiency, and user experience. Recently, Feizi et al.~~\citep{feizi2023online} and Hajiaghayi et al.~~\citep{hajiaghayi2024ad} outlined a retrieve-then-generate paradigm that decouples retrieval and generation, offering lightweight ad insertion and payment determination. However, current retrieval relies solely on text embedding similarity, which may lead to commercial misinterpretation and issues such as repetitive insertions. In this paper, we propose LERA, a two-stage retrieve-then-generate auction framework tailored for LLM chatbots. In the first stage, embedding-based coarse filtering pre-selects a small set of candidate advertisers. In the second stage, the LLM itself is queried with a carefully designed prompt to produce logits over candidates, which serve as refined organic relevance scores. These scores are combined with bids, and a critical-value payment rule accounts for both the coarse-filtering and fine-ranking thresholds, ensuring truthfulness for utility-maximizing advertisers. The framework naturally extends to multiple ad insertions within dynamic dialogue flows and long responses. Experiments on a synthetic advertiser-query benchmark show that LERA substantially improves ad selection accuracy and insertion diversity while incurring only controllable latency overhead.

Haoran Sun, Xinrui Song, Xinyu Zhang +7

Peking University · Shandong University · Alibaba Group

Retrieval-Augmented Generation Mechanism Design Online Advertising LLM Reranking Conversational Agents

Ask PDF