From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRL

Wenyue Hua; Zachary Huang; Tyler Payne; Safoora Yousefi; Saleema Amershi; Asli Celikyilmaz

cs.AIAug 13, 2026

From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRL

Authors: Wenyue Hua, Zachary Huang, Tyler Payne, Safoora Yousefi, Saleema Amershi, Asli Celikyilmaz

Organizations: Microsoft Research, AI Frontiers

Abstract

AI agents increasingly act on their users' behalf, handling tasks such as scheduling meetings, comparing offers, and haggling over prices. These principal-driven tasks routinely place the agent across from a counterpart (another user's agent, a seller, a recruiter) whose goals may conflict with its principal's. Yet the dispositions that make an assistant pleasant can make it a poor delegate: a friendly, helpful frontier model may disclose its principal's private information unprompted and concede at the first sign of resistance. We present SocialRL, a general recipe that trains social reasoning directly, and apply it to a 4B model across six domains: Deal-or-No-Deal, CaSiNo, Craigslist, Job Interview, Calendar, and Marketplace. Every domain is trained in-domain under the same recipe, and every policy is evaluated on all six. We find that (1) in-domain training reaches the frontier: on held-out scenarios the 4B matches or exceeds the GPT-5 family per domain, closing 73-122% of the baseline-to-frontier gap on the negotiation games, with 78% of buyer openings anchoring below target versus 3% untrained; (2) cross-domain transfer follows game structure: structurally paired games lift each other, a broad multi-issue donor lifts nearly all domains, and structurally isolated games transfer nothing; (3) guided by this transfer structure, two strategies, cascade RL and multi-teacher on-policy distillation (OPD), consolidate the per-domain specialists into a single unified 4B that reaches 0.627 average utility across all six environments, matching or exceeding GPT-4.1 (0.625), GPT-5.1 (0.619), and GPT-5.2 (0.613); (4) an explicit theory-of-mind scaffold helps only through training: distilling the ToM trace, rather than actions alone, lifts utility on every environment and generalizes better across them, and of the two ToM skills, only next-action prediction predicts negotiation outcomes.

Figures & tables

Figure 1: SocialRL overview: interaction environments, event-based agent interface, and decoupled training infrastructure. (a) Example social-reasoning environments for delegated coordination and negotiation. (b) An event-based, agent-independent interface in which a stateful environment communicates with agents through private channels of observations, notifications, and actions. (c) A decoupled training stack where the rollout proxy connects the environment and agent harness to inference and training, enabling interchangeable components and supporting both reinforcement learning and distillation.

Figure 2: Environment and agent architecture. A stateful environment communicates with black-box agents through per-agent asynchronous channels carrying structured observations, notifications, and actions. The same interface supports LLM agents, external coding agents, humans, and scripted policies. Model inference is isolated behind an OpenAI-compatible interface, while the event stream supports agent execution, human play, observability, replay, and trajectory collection.

Figure 3: Decoupled training architecture. The agent runtime executes independently of the trainer and sends standard model requests through a rollout proxy. The proxy forwards requests to the inference engine while recording the authentic model inputs and outputs associated with each episode. Completed trajectories are written to a shared Parquet buffer consumed by interchangeable training backends, and updated policy weights are synchronized back to the inference engine.

Domain	Structure	Issues	Private information	Roles
Deal-or-No-Deal	multi-issue allocation	3 item types	per-item values	symmetric
CaSiNo	multi-issue allocation	3 resources	priority order	symmetric
Craigslist	single-issue price	price	target / listing price	asymmetric
Job Interview	multi-issue contract	5 issues	utility weights	asymmetric
Calendar	slot coordination	time slots	slot preferences	asymmetric
Marketplace	single-issue price	price	reservation price	asymmetric

Table 1: The six social-reasoning environments. The suite spans heterogeneous negotiation and coordination structures while sharing a common requirement to reason strategically about a counterpart with private objectives.

	DnD	CaSiNo	Craigslist	Job Int.	Calendar	Mktplace	Avg.
Base 4B	$0.583\pm 0.001$	$0.476\pm 0.014$	$0.318\pm 0.012$	$0.479\pm 0.032$	$0.301\pm 0.017$	$0.174\pm 0.026$	0.389
Domain-trained 4B	$0.656\pm 0.008$	$0.503\pm 0.007$	$0.583\pm 0.008$	$0.594\pm 0.007$	$0.540\pm 0.017$	$0.838\pm 0.013$	0.619
GPT-4.1	0.653	0.491	0.540	0.588	0.673	0.804	0.625
GPT-5.1	0.671	0.499	0.607	0.596	0.573	0.767	0.619
GPT-5.2	0.663	0.488	0.577	0.579	0.643	0.727	0.613
GPT-5.5	0.665	0.559	0.746	0.590	0.702	0.985	0.708

Table 2: In-domain specialist training. Base and domain-trained 4B scores are reported as mean

\pm

standard deviation across repeated evaluations; GPT baselines are reported as mean scores. Each domain-trained checkpoint is evaluated on its training environment. The six 4B specialists average 0.619 utility, placing their aggregate performance in the range of GPT-4.1, GPT-5.1, and GPT-5.2.

Donor	DnD	CaSiNo	Craigslist	Job Int.	Calendar	Mktplace	Out-tr.
Baseline	$0.583{\pm}.001$	$0.476{\pm}.014$	$0.318{\pm}.012$	$0.479{\pm}.032$	$0.301{\pm}.012$	$0.174{\pm}.003$	—
Deal-or-No-Deal	$\mathbf{0.656{\pm}.008}^{\star}$	$0.491{\pm}.012$	$0.282{\pm}.028$	$0.481{\pm}.016$	$0.370{\pm}.009$	$0.150{\pm}.003$	$+0.005$
CaSiNo	$0.597{\pm}.019$	$\mathbf{0.503{\pm}.007}^{\star}$	$0.309{\pm}.004$	$0.463{\pm}.016$	$0.331{\pm}.010$	$0.168{\pm}.022$	$+0.003$
Craigslist †	$0.564{\pm}.023$	$0.494{\pm}.007$	$\mathbf{0.583{\pm}.008}^{\star}$	$0.447{\pm}.011$	$0.266{\pm}.023$	$0.491{\pm}.007$	$\mathbf{+0.050}$
Job Interview	$0.610{\pm}.017$	$0.511{\pm}.004$	$0.328{\pm}.010$	$\mathbf{0.594{\pm}.007}^{\star}$	$0.373{\pm}.022$	$0.161{\pm}.011$	$+0.026$
Calendar	$0.607{\pm}.010$	$0.474{\pm}.007$	$0.167{\pm}.013$	$0.311{\pm}.013$	$\mathbf{0.540{\pm}.017}^{\star}$	$0.172{\pm}.008$	$-0.060$

Table 3: Cross-environment transfer. Each row evaluates a policy trained on one donor environment across all six domains (

\star

denotes the training domain). Out-transfer is the mean change from the base 4B policy over the five off-domain environments. † Craigslist uses SFT+PPO. Reported uncertainties are standard deviations across three SGLang benchmark runs where available.

Stage completed	Step	DnD	CaSiNo	Craigslist	Job Int.	Calendar	Mktplace	Mean seen
Calendar	100	—	—	0.614	—	0.621	—	0.617
CaSiNo	40	—	0.504	0.517	—	0.733	—	0.585
Deal-or-No-Deal	100	0.618	0.498	0.489	—	0.758	—	0.591
Craigslist	40	0.628	0.477	0.544	—	0.729	—	0.595
Marketplace	140	0.628	0.483	0.564	—	0.704	0.842	0.644
Job Interview	60	0.620	0.513	0.570	0.529	0.692	0.823	0.625

Table 4: Stage-by-stage evolution of the transfer-aware cascade. Each row reports the checkpoint selected after optimizing the environment in the first column; bold denotes the environment optimized at that stage. Craigslist is evaluated from the beginning because the cascade is initialized from its SFT checkpoint. Dashes denote environments not yet included in checkpoint selection. Mean seen averages the environments tracked at that stage and is therefore not directly comparable across rows as the set of environments expands. These are within-cascade checkpoint-selection evaluations; Table 5 reports the final repeated six-environment benchmark.

Environment	Transfer-aware	Random	Anti-transfer
Deal-or-No-Deal	$0.617\pm 0.008$	$0.604\pm 0.012$	$0.622\pm 0.018$
CaSiNo	$0.482\pm 0.019$	$0.465\pm 0.016$	$0.443\pm 0.023$
Craigslist	$0.580\pm 0.007$	$\mathbf{0.637\pm 0.022}$	$0.497\pm 0.018$
Job Interview	$0.538\pm 0.012$	$0.478\pm 0.030$	$0.471\pm 0.009$
Calendar	$\mathbf{0.742\pm 0.037}$	$0.541\pm 0.021$	$0.641\pm 0.011$
Marketplace	$0.803\pm 0.008$	$0.783\pm 0.024$	$0.698\pm 0.022$

Table 5: Cascade RL is sensitive to environment ordering. The transfer-aware curriculum follows Calendar

\rightarrow

CaSiNo

\rightarrow

DnD

\rightarrow

Craigslist

\rightarrow

Marketplace

\rightarrow

Job Interview; the random-order control follows Eq. 12 ; and the anti-transfer control follows Eq. 13 . Each final evaluation uses

10

scenarios

\times

3

opponents

\times

5

trials (

150

games per environment). Transfer-aware scores are mean

\pm

standard deviation over two independent evaluations, and random-order scores are mean

\pm

standard deviation over repeated evaluations. Anti-transfer performs way worse than both Transfer-aware as well as Random-order. The transfer-aware unified 4B policy achieves an Avg-6 of

0.627

, in the performance range of GPT-4.1, GPT-5.1, and GPT-5.2.

Configuration	Avg-6 $\uparrow$	Gap closure $\uparrow$	Clipped gap closure $\uparrow$
Gap-closed, w/o CaSiNo	$\mathbf{0.597\pm 0.015}$	$\mathbf{92.6\pm 11.3}$ %	$\mathbf{84.5\pm 6.3}$ %
Equal mix, w/o CaSiNo	$0.594\pm 0.020$	$84.4\pm 15.6$ %	$79.7\pm 12.6$ %
Equal mix, w/ CaSiNo	$0.595\pm 0.015$	$82.2\pm 9.2$ %	$77.6\pm 5.8$ %
Gap-closed, w/ CaSiNo	$0.578\pm 0.012$	$71.3\pm 8.7$ %	$70.7\pm 7.9$ %
Student initialization	0.460	—	—
Teacher average	0.619	—	—

Table 6: MOPD. All configurations use reverse-KL distillation without position weighting and a learning rate of

10^{-5}

. Scores are mean

\pm

standard deviation over five independent evaluation runs. The gap-closed curriculum without the saturated CaSiNo teacher performs best under all three aggregate metrics.

Environment	Unified 4B	Initialization	Teacher	Gap closed
Deal-or-No-Deal	$\mathbf{0.664\pm 0.025}$	0.586	$0.656\pm 0.008$	$\mathbf{111\%}$
Craigslist	$0.556\pm 0.019$	0.397	$0.583\pm 0.007$	$85\%$
CaSiNo †	$0.478\pm 0.019$	0.502	$0.503\pm 0.008$	—
Job Interview	$0.551\pm 0.023$	0.456	$0.594\pm 0.007$	$69\%$
Calendar	$\mathbf{0.564\pm 0.050}$	0.375	$0.540\pm 0.017$	$\mathbf{115\%}$
Marketplace	$0.771\pm 0.028$	0.444	$0.838\pm 0.013$	$83\%$

Table 7: Per-environment performance of the best MOPD model. We report the gap-closed curriculum without CaSiNo at step 60. Unified-model scores are mean

\pm

standard deviation over five independent evaluation runs. Gap closure measures the fraction of each specialist’s advantage over the initialization recovered by the unified model. † CaSiNo is excluded from MOPD training and gap-closure aggregation because its teacher–initialization gap is only

0.001

. ∗ Mean gap closure over the remaining five environments.

Environment	Base 4B	+ToM prompt	per-env	mixed	per-env	mixed	GPT-5.2
			Normal SFT		ExpToM SFT
Deal-or-No-Deal	0.589	0.575	0.580	0.587	0.596	0.608	0.655
CaSiNo	0.454	0.391	0.492	0.480	0.512	0.498	0.475
Craigslist	0.295	0.225	0.445	0.409	0.552	0.478	0.511
Job Interview	0.476	0.221	0.482	0.503	0.522	0.517	0.626
Average	0.454	0.353	0.500	0.495	0.546	0.525	0.567

Table 8: Explicit theory-of-mind supervision improves negotiation. Normalized utility for the base model, prompt-only ToM, ordinary SFT, and explicit ToM SFT. ExpToM distills Infer

\rightarrow

Act

\rightarrow

Anticipate traces from GPT-5.2. Prompting the scaffold alone hurts the base 4B, while training on the same reasoning structure improves over ordinary SFT under both per-environment and mixed training.

Stage	Open	2nd	3rd	4th	% open $<$ target
Untrained	$+0.04$	$+0.35$	$+0.41$	$+0.45$	3%
SFT	$-1.21$	$-0.30$	$-0.08$	$-0.06$	71%
SFT $+$ PPO	$-1.48$	$-0.73$	$-0.22$	$+0.01$	78%

Table 9: Craigslist buyer concession curves. Offer price is normalized by the listing-target gap:

0

denotes the buyer’s target,

1

the listing price, and negative values offers below target. SFT introduces below-target anchoring; PPO produces a lower initial anchor and a slower subsequent concession trajectory.

Env	Stage	Turns	Polite	Firm
CaSiNo	untrained	2.5	16%	1%
	SFT	3.4	10%	4%
	PPO	3.4	59%	11%
DnD	untrained	2.9	8%	2%
	SFT	3.1	15%	23%
	PPO	3.2	36%	2%

Table 10: Interaction behavior across the training ladder. Turns denotes turns per game; public tok is the token length of the opponent-facing message; total tok includes the full generation associated with each turn; <think> is the fraction of turns containing an explicit reasoning block; polite and firm denote the fractions of public messages containing courteous and commitment language, respectively. Tokens are measured with the Qwen3-4B tokenizer.

Explore similar work

Oct 5, 2026

Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellers

LLM agents are starting to own the full customer experience. Soon, LLMs may be selling and buying on behalf of companies and customers respectively. Small models are more cost-efficient at scale, but can reinforcement learning train them into competent sellers? We train four Gemma 4 checkpoints (2.3B to 31B effective parameters) with GRPO on a programmatic utility reward for bilateral multi-issue bargaining, and evaluate every arm on the same 1,152 negotiations against two frontier buyers it never saw in training. With the same learning rate (

10^{-6}

) for every size, the gain of the RL model over its base rises from

+0.001

at 2.3B to

+0.078

at 31B. Each size was trained once and the two smallest checkpoints use a different architecture, so we fit no scaling law. Tripling the learning rate, with the same or fewer training steps, improves on the shared rate at every size by

+0.032

(2.3B) to

+0.081

(4.5B). In exploratory comparisons with two frontier models run as sellers, the 12B seller trained at the tripled rate scores above both, though its untrained base already scores as high as they do. The 4.5B seller at that rate shows no detectable difference from either and fits on one 48 GB GPU. A further 2.3B arm at ten times the shared rate raises pooled score, but its gain concentrates on the evaluation buyer that shares a model family with the training pool. These results suggest tuning the learning rate before concluding that a small model cannot learn to negotiate, and testing against buyers from more than one model family.

Pedro Tabacof, Sagar Joglekar

Fin AI Research

RL for Language Models Multi-Agent Negotiation Small Language Models Language Model Scaling

Ask PDF

Feb 12, 2026

Choose Your Agent: Tradeoffs in Adopting AI Advisors, Coaches, and Delegates in Multi-Party Negotiation

As AI usage becomes more prevalent in social contexts, understanding agent-user interaction is critical to designing systems that imp rove both individual and group outcomes. We present an online behavioral experiment (N=243) in which participants play three multi-tu rn bargaining games in groups of three. Each game, presented in randomized order, grants access to a single LLM assistance modality: proactive recommendations from an Advisor, reactive feedback from a Coach, or autonomous execution by a Delegate. All three modalitie s are powered by an LLM with super-human performance within this negotiation setting. On each turn, participants privately decide whe ther to act manually or use the AI modality available in that game. We document a preference-performance misalignment: participants s trongly prefer the higher-control Advisor (44%) over the Delegate (19%), yet groups only significantly increase collective surplus un der Delegate access. Adjusting for voluntary non-compliance, delegating to the AI yields suggestive individual welfare gains, roughly 1.5x the intent-to-treat estimate. A mechanism analysis traces this gap to a human filter: AI-generated proposals create more joint surplus than manual proposals across all conditions, but in the Advisor and Coach modes users modify, override, or ignore the AI's su ggestions, reverting toward human-baseline trade patterns. The Delegate advantage arises not from a different AI capability but from bypassing this filtering step altogether. Realizing these welfare gains depends not only on model capability, but on the interaction structure through which that capability is delivered. We argue that assistance modalities should be designed as mechanisms with endog enous participation; adoption-compatible interaction rules are a prerequisite to improving welfare with automated assistance.

Kehang Zhu, Nithum Thain, Vivian Tsai +2

Harvard University, United States · Google DeepMind, Canada · Google DeepMind, United States +1

Automated Negotiation Human-AI Interaction Human-AI Decision Making Human-AI Collaboration

Ask PDF

Aug 6, 2026

Enhancing Social Intelligence in LLMs with Hierarchical Reasoning and Utterance-Level Goal Rewarding

Large language models (LLMs) excel in structured tasks but struggle with dynamic social interactions, where success requires long-term goal coordination and rapid adaptation. Current methods often apply uniform goal-based rewards to every utterance, overlooking the specificity of objectives at each dialogue turn and failing to account for the rationale of potential strategies. Inspired by the Theory of Planned Behavior, we propose the Think-Strategy-Response (TSR) framework, which decomposes social dialogue into two hierarchical stages: high-level strategic planning and low-level linguistic execution. To optimize TSR, we introduce Linearized Hierarchical Reinforcement Learning with Variance-Gated Rewards (LHRL-VGR), a novel algorithm that dynamically routes rewards - balancing goal completion and strategy adherence - based on the variance of goal achievement scores. Experiments on the SOTOPIA benchmark show that our approach fine-tunes a Qwen2.5-7B agent to surpass the GPT-4o baseline by 7.32% in goal completion success, demonstrating state-of-the-art performance in multi-agent social negotiation tasks.

Xiaofeng Wang, Kakam Chong, Shuai Xiao +9

Independent Researcher · Alibaba Group · Shanghai Jiao Tong University

Reward Shaping Hierarchical RL Goal-Conditioned RL RL for LLM Agents

Ask PDF