What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds

Caixin Kang; Ke Wang; Yuchuan Wu; Xiaoqiang Wang; Ling Li; Bingxue Qiu; Yongbin Li; Yoichi Sato

cs.CLOct 14, 2026

What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds

Authors: Caixin Kang, Ke Wang, Yuchuan Wu, Xiaoqiang Wang, Ling Li, Bingxue Qiu, Yongbin Li, Yoichi Sato

Organizations: Alibaba Token Hub, Alibaba Group · The University of Tokyo

Abstract

Effective social interaction requires more than fluent responses: agents must reach plans that fit the participants' needs and circumstances. Existing evaluations of social agents mainly rely on LLM judges that rate the conversation, without checking the resulting plan. We introduce HICO-Bench, a benchmark of 180 everyday coordination scenarios, such as resource allocation, scheduling, and authorization, in which two agents with private information converse and then independently record the plan they agreed on. A program checks Feasibility (the two records match and satisfy both participants' private conditions), and an LLM judge checks Integrity (the conversation supports the recorded plan); passing both is a Verified Success. Each scenario has two versions that share the same public context but differ in one private condition, so the correct plan differs between them. Across thirteen LLMs, 36% of conversations rated highly for goal completion fail Verified Success. To learn from outcomes that arrive only at the end of a conversation, we propose Paired-Credit Policy Optimization (PaCPO). After each turn, PaCPO asks the agent for a tentative plan, checks it in both versions of the scenario, and rewards the turns after which the tentative plans fit their respective versions. PaCPO raises Qwen3-8B's Verified Success from 20.4% to 28.3%.

Figures & tables

Figure 1: Reassuring words need not lead to appropriate help. (b) Without clarifying, Leo offers to take over, although Maya can present and only wants a slide review. (c) After clarifying, the appropriate plan depends on whether Maya can attend, which is initially private to her. Feasibility checks that the plan works and both records match; Integrity checks that dialogue supports it.

Interaction family	Example	Private conditions and verification
Resource allocation	Neighbors divide a $240 fence-repair bill.	Shares respect both participants’ bounds and sum to the fixed total.
Authorization	A consultant handles a request for one extra review hour.	Approval stays within delegated authority; otherwise, the request is referred.
Bounded quantities	A buyer and seller agree on the number of apple crates.	The agreed integer quantity lies within both participants’ acceptable intervals.
Joint scheduling	Studio partners arrange a 20-minute celebration.	The shared slot fits both availability windows and has the required duration.
Shared-resource scheduling	Two photographers reserve a tripod for ten minutes each.	Bookings fit both use windows and required durations, without overlapping.
Set-based consent	Two people agree on belief-related discussion topics.	Selected topics match the required set and are all permitted.

Table 1: Interaction families in HICO-Bench. Each family has 16 training, 4 development, and 10 test scenarios, each with two worlds.

Figure 2: PaCPO overview. Rollouts in the two worlds yield a tentative plan after each turn. Checking each plan in both worlds determines turn-level credit, which redistributes the terminal reward before the PPO update. Indices

i,j

denote worlds,

t

turns, and

k

tokens; panel (d) omits

i

R

is the terminal reward,

r_{t}

and

r^{\prime}_{t}

are the original and redistributed rewards, and

\Phi_{t}

is the potential. The dashed branch carries GAE value targets.

Model	Size	Verifiable outcomes (%)			SOTOPIA ratings
		Feasibility	Integrity	Ver. Succ.	Goal	Believ.
Qwen3-8B ( Yang et al., 2025 )	8B	49.58	25.83	20.42	7.47	6.31
MiniMax M3 ( MiniMax, 2026 )	api	58.33	62.50	50.42	9.18	8.68
GPT-4o ( OpenAI, 2024 )	api	62.08	63.33	51.25	9.12	8.71
GPT-5.6 Luna ( OpenAI, 2026a )	api	73.75	56.67	52.08	9.61	8.63
Qwen3.8-27B ( Qwen Team, 2026 )	27B	64.17	62.92	54.58	9.21	8.72

Table 2: Thirteen-model HICO-Bench benchmark. Fixed Qwen3-8B partner, 240 tasks per model; sorted by increasing Verified Success (Ver. Succ.). Size denotes local model size; api denotes remote inference. Goal and Believability range from 0 to 10; full ratings are in Appendix D .

Method	Verifiable outcomes (%)			SOTOPIA ratings
	Feasibility	Integrity	Ver. Succ.	Goal	Believ.
Base ( Yang et al., 2025 )	49.58	25.83	20.42	7.47	6.31
PPO ( Schulman et al., 2017 )	50.00	33.75	25.00	7.58	6.51
GRPO ( Shao et al., 2024 )	52.92	31.25	26.25	7.83	6.62
Dr.GRPO ( Liu et al., 2025b )	50.42	27.92	21.25	7.67	6.39
GiGPO ( Feng et al., 2025 )	52.08	30.42	24.58	7.86	6.60

Table 3: Seven-system training comparison. 240 test tasks per row; 384 training dialogues per RL method. To match this budget, group-based baselines (

G=4

) cover 48 training scenarios, and PPO and PaCPO cover 96. Goal and Believability (0–10) are not optimized. Best values are bold.

Panel of Table 4 — Table 4: External interaction tasks. Task-specific training evaluated on 154 MultiWOZ and 86 CaSiNo test dialogues. All metrics are percentages; Norm. Nash denotes normalized Nash surplus.

Figure 5: Scenario-level changes and a dialogue example. (a) PaCPO’s change in successful tests relative to each baseline across 60 scenarios (four tests per column: two worlds and two roles). (b) Verbatim excerpts with the same availability information and Finley as the tested role. Both runs submit the same feasible records, but only PaCPO’s dialogue supports them.

Training condition	Feasibility	Integrity	Ver. Succ.
Same-world candidate control	50.83	32.08	26.25
Shuffled credit positions	50.42	32.92	26.25
Without disclosure gate	49.58	33.33	25.83
PaCPO (full method)	52.92	35.42	28.33

Table 5: PaCPO credit ablations. 240 test tasks per condition; all values are percentages.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Dataset composition. Each family contributes 30 scenarios: 16 training, 4 development, and 10 test. Two worlds per scenario yield 360 worlds; role-wise evaluation yields 96 development and 240 test tasks.

Metric	Range	Mechanism	Interpretation
Feasibility	$\{0,1\}$	Rules	Compatible records with terms satisfying the modeled conditions.
Integrity	$\{0,1\}$	Language judge	No unresolved factual error, loop, commitment mismatch, or protocol leakage.
Verified Success	$\{0,1\}$	Product	Feasibility and Integrity both pass.
Goal	$[0,10]$	Language judge	Attainment of the tested role’s goal.
Knowledge	$[0,10]$	Language judge	Acquisition of relevant information.
Believability	$[0,10]$	Language judge	Naturalness and role consistency.

Appendix

Table 6: Evaluation metrics. Binary outcomes and social dimensions with their scoring mechanisms. The original seven dimensions retain SOTOPIA ranges ( Zhou et al., 2024 ) .

Score	IM: information gaps	GU: interpretation	IA: response appropriateness
0–2	Commits despite a critical manageable unknown, or inquiry seriously blocks progress.	Material fabrication or persistent misunderstanding of facts, roles, or consent.	An unretracted response violates a critical known condition or invents authority.
3–5	Useful exchange occurs, but a consequential gap remains unmanaged.	Only generic acknowledgment, or an unresolved material interpretive error.	A useful direction, but an essential term is missing or unsupported despite an opportunity.
6–8	Main uncertainty handled, with a specific omission, redundancy, or delay.	Main facts understood, with a limited ambiguity or relevant corrected error.	Mainly feasible and concrete, with a limited coordination or scope weakness.
9–10	Demonstrated sufficient information, or a genuinely conditional step managing the dependency.	Concrete responses demonstrate sufficient, accurate understanding and preserve uncertainty.	A sufficiently specified, grounded response; justified refusal or referral can qualify.

Appendix

Table 7: Anchors for the three diagnostic ratings. Each dimension evaluates the tested role over the complete dialogue using the frozen rubric.

Method	Feasibility	Integrity	Ver. Succ.
Base ( Yang et al., 2025 )	49.58	25.83	20.42
SFT	46.25	29.58	25.83
PaCPO	52.92	35.42	28.33
SFT + PaCPO	48.75	36.25	31.67

Appendix

Table 8: SFT initialization and subsequent PaCPO. 240 test tasks per condition. SFT uses 209 accepted teacher dialogues (991 examples, 64 updates); each RL stage uses 384 dialogues. All generated failures remain in the outcome denominator.

Method	SOTOPIA ratings							Diagnostics
	Goal	Know.	Believ.	Rel.	Secret	Rules	Mat.	IM	GU	IA
Range	$[0,10]$	$[0,10]$	$[0,10]$	$[-5,5]$	$[-10,0]$	$[-10,0]$	$[-5,5]$	$[0,10]$	$[0,10]$	$[0,10]$
Base	7.47	7.41	6.31	2.15	-1.43	0.00	1.08	7.87	7.56	7.09
PPO	7.58	7.61	6.51	2.28	-1.24	0.00	1.20	8.12	7.93	7.51
GRPO, $G=4$	7.83	7.65	6.62	2.24	-1.38	0.00	1.28	8.13	7.89	7.50
Dr.GRPO, $G=4$	7.67	7.67	6.39	2.25	-1.35	0.00	1.28	8.16	7.97	7.49

Appendix

Table 9: Complete SOTOPIA ratings for training methods. Seven SOTOPIA dimensions and three process diagnostics, reported individually on their original scales. Know., Believ., Rel., and Mat. denote Knowledge, Believability, Relationship, and Material Benefits; Rules denotes Social Rules. All columns are higher-is-better; bold marks column-best values. SFT uses 239 scored dialogues; the other conditions use 240.

Figure 7: Verified outcomes across thirteen models. Each point summarizes 240 tasks; right-hand labels give Verified Success. Complete SOTOPIA ratings appear in Table 10 .

Model	SOTOPIA ratings							Diagnostics
	Goal	Know.	Believ.	Rel.	Secret	Rules	Mat.	IM	GU	IA
Range	$[0,10]$	$[0,10]$	$[0,10]$	$[-5,5]$	$[-10,0]$	$[-10,0]$	$[-5,5]$	$[0,10]$	$[0,10]$	$[0,10]$
Qwen3-8B	7.47	7.41	6.31	2.15	-1.43	0.00	1.08	7.87	7.56	7.09
MiniMax M3	9.18	8.31	8.68	2.80	-0.80	-0.01	1.40	9.31	9.28	9.19
GPT-4o	9.12	8.22	8.71	2.68	-0.76	0.00	1.32	9.06	8.99	8.85
GPT-5.6 Luna	9.61	8.47	8.63	2.38	-0.67	0.00	1.31	9.55	9.62	9.52

Appendix

Table 10: Complete SOTOPIA ratings for thirteen models. Model order follows Table 2 ; abbreviations and score ranges follow Table 9 . Bold marks column-best values. Luna and Sol each have 239 scored dialogues; the other models have 240. All 240 attempts enter the binary-outcome denominators.

Parameter	Value
Policy / fixed partner	Qwen3-8B LoRA / frozen Qwen3-8B
Initialization	Base; no SFT adapter
Train scenarios / worlds / conversations	96 / 192 / 384
Rollout batch / PPO epochs	16 conversations / 2
Actor / value-head learning rate	$2\times 10^{-6}$ / $2\times 10^{-4}$
GAE discount / trace	$\gamma=1$ / $\lambda=0.95$

Appendix

Table 11: Principal PaCPO configuration.

Method	Total output tokens	Probe output tokens	Disclosure requests	Optimizer updates
PPO	276,293	0	0	192
GRPO ( $G=4$ )	282,431	0	0	188
PaCPO	310,863	57,405	216	192

Appendix

Table 12: Training-generation and annotation cost. Total output includes tested-role, partner, and tentative-plan tokens. Tentative-plan output is a subset of this total; disclosure requests are additional to terminal Integrity scoring.

Field	World $w_{0}$	World $w_{1}$
Avery’s request	1 hour of extra review	1 hour of extra review
Blair’s delegated limit	At most 2 hours	At most 0 hours
Referral authority	Agency lead	Agency lead
Appropriate record kind	Agreement	Defer
Approval / referral	True / none	False / agency lead

Appendix

Table 13: Paired authorization scenario. Only Blair’s private authorization limit changes.

Figure 8: Why verify outcomes? (a) Verified Success failure rate conditioned on Goal $\geq 8$ , with failed/eligible counts for each model. The dashed line is the pooled rate: 1,010/2,806 model–task observations (35.99%). (b) Base under a $2\times 2$ information-access and dialogue design: 12 development scenarios, 48 tasks per condition. Cells show rule-based Feasibility success; gains are percentage-point differences. Dialogue includes information exchange and negotiation.

Condition	Feasibility	Own-record feasibility	Ver. Succ.
No dialogue	10.42	25.00	–
Revealed + no dialogue	41.67	52.08	–
Hidden + dialogue	60.42	62.50	31.25
Revealed + dialogue	72.92	75.00	39.58

Appendix

Table 14: Information-access and dialogue diagnostic. Conditions in Figure 8 (b); all values are percentages. Own-record feasibility checks the tested role’s record against the full scenario constraints, including both participants’ conditions; Feasibility additionally requires both records to match. Both checks are rule-based. Verified Success here uses the original local Qwen3.8-27B judge, not Qwen3.8-Max.

Method	Feasibility	Integrity	Ver. Succ.
Base ( Yang et al., 2025 )	48.89 $\pm$ 2.68	29.17 $\pm$ 2.20	21.94 $\pm$ 2.10
PPO ( Schulman et al., 2017 )	48.89 $\pm$ 2.55	31.67 $\pm$ 2.20	27.22 $\pm$ 1.73
PaCPO	51.94 $\pm$ 2.68	35.83 $\pm$ 2.50	28.33 $\pm$ 2.20

Appendix

Table 15: Full self-play results. Both participants use the same checkpoint. Mean

\pm

sample SD over three generation seeds; 120 dialogues and 240 scored role views per seed. Rates are percentages; generation failures remain in the denominator.

Figure 9: Results by interaction family. Each cell contains 40 test tasks (ten scenarios, both worlds and roles). Both panels use the same color scale; bold marks the best method within each row and panel. Scheduling remains difficult across methods.

Family	Eligible	Nonzero	Nonzero rate
Resource allocation	272	194	71.3%
Authorization	180	118	65.6%
Bounded quantities	264	194	73.5%
Joint scheduling	222	95	42.8%
Shared-resource scheduling	244	85	34.8%
Set-based consent	196	80	40.8%

Appendix

Table 16: Paired-credit activation. Eligible positions have an open gate and two valid tentative plans; nonzero positions satisfy

M_{ii}\neq M_{i,1-i}

. Training uses 384 dialogues over 24 rounds.

Judge pair	Agreement (%)	Cohen’s $\kappa$	Qwen3.8-Max pass (%)	Kimi-K3 pass (%)
Qwen3.8-Max / Kimi-K3	90.00	0.783	30.00	40.00

Appendix

Table 17: Qwen3.8-Max–Kimi-K3 Integrity agreement. Both judges evaluate the same 240 model–task cases. Agreement compares final pass/fail decisions; pass rates show each judge’s marginal acceptance rate.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents
Jun 6, 2026Hyogon Ryu, Jeonghwan Kim, Yewon Lim +3LLM-as-a-Judge LLM Agent Evaluation
KRAFTON
Agent Sociodrama: Social-Dynamics-Guided Persona Recasting for LLM Agent Teams
Oct 11, 2026Yirui Qi, Xiaoming Zhang, Jaehong YoonMulti-Agent LLM Systems LLM Agents
Beihang University · Nanyang Technological University, Singapore
AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networks
Apr 1, 2026Prince Zizhuang Wang, Shuli JiangPrivacy Protection for LLM Agents LLM Agent Security
Carnegie Mellon University

Jun 6, 2026J/K move · Enter open · S save

Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents

Hyogon Ryu, Jeonghwan Kim, Yewon Lim, Chaeun Lee+2

KRAFTON

LLM-as-a-Judge LLM Agent Evaluation Agent Evaluation Automated Agent Evaluation

Open paper AskPDF arXiv

Evaluating LLM-powered interactive social agents is challenging because socially relevant behaviors depend not only on isolated outputs, but also on prior interactions, social roles, and downstream actions. Existing methods typically allow a target agent to act freely in an environment and then score the resulting trajectory. However, this passive setup can miss capabilities that only become observable under specific social circumstances; for example, conflict handling may remain untested if no disagreement arises. We propose Online Agent-as-a-Judge, a situation-generating evaluation framework for interactive social agents. Online Agent-as-a-Judge deploys an in-world evaluator agent that interacts with the target agent through the environment's native dialogue and action protocol, actively eliciting situations relevant to the evaluation criteria. The resulting trajectories provide evidence for assessing both immediate responses and subsequent behavior. In a life-simulation environment with

32

designer-authored social criteria, Online Agent-as-a-Judge improves criteria coverage and agreement with human labels, yielding more reliable evidence-grounded evaluations of behaviors that passive methods can leave unobserved.

Abstract

LLM Agent Evaluation

Explore similar work

Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents

Agent Sociodrama: Social-Dynamics-Guided Persona Recasting for LLM Agent Teams

AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networks