SocialBuddy: Tailoring Search Agent for Social Scenarios

Mingxuan Li; Yirong Mao; FaZhan Zhang; Haibiao Yao; Runze Hu; Wenhui Que

cs.SIAug 28, 2026

SocialBuddy: Tailoring Search Agent for Social Scenarios

Authors: Mingxuan Li, Yirong Mao, FaZhan Zhang, Haibiao Yao, Runze Hu, Wenhui Que

Organizations: WeChat, Tencent Inc, China · School of Information and Electronics, Beijing Institute of Technology, China · School of Computer Science and Technology, University of Science and Technology of China, China

Abstract

In the era of digital social interaction, searching friends' posts from massive social streams has become a fundamental user need. However, while modern agentic search frameworks have achieved remarkable success in conventional retrieval tasks, they break down when confronted with heterogeneous user queries and multi-dimensional social feeds, resulting in severe performance degradation in complex social search. To bridge this gap, we introduce SocialBuddy, the first agentic search framework tailored for social scenarios. Specifically, we construct SocialEnv, the first large-scale simulated environment for social search. Powered by an automated data and trajectory synthesis pipeline, SocialEnv includes 200K user profiles, 10 million social posts, and 50K reasoning trajectories, establishing a solid foundation for the development of social search agents. To tackle the credit assignment dilemma caused by sparse rewards in social search, we design SocialPO, a hybrid-granularity optimization framework. It macroscopically reinforces successful reasoning paths via multi-dimensional rewards, while microscopically rectifying deviated trajectories through fine-grained prefix truncation and token-level supervision. This hybrid-granularity design delivers multi-scale guidance in complex long-sequence scenarios. Finally, we construct SocialSearch Benchmark to provide a quantitative evaluation scheme for assessing the social search capabilities of SocialBuddy. Extensive experiments demonstrate that SocialBuddy-35B surpasses significantly larger frontier LLMs. Code and dataset will be released upon article acceptance.

Figures & tables

Figure 1: A comparison of social search approaches. Left: Standard LLMs and general search agents fail due to data access barriers and the mismatch between text retrieval and multi-dimensional social constraints. Right: SocialBuddy employs social-adapted tools to decompose complex queries and perform relational reasoning over structured metadata.

Figure 2: The Overall framework of SocialBuddy with three main phases. Phase 1: Building the interactive SocialEnv; Phase 2: Curating multi-intent query-trajectory pairs; and Phase 3: Optimizing SocialBuddy via the SocialPO algorithm.

Type	Model	Size	Easy Set				Hard Set				Avg. Overall
Type	Model	Size	SR	Prec.	Recall	EM	SR	Prec.	Recall	EM	Prec.	Recall	EM
API-based	Claude-4.5-Sonnet	N/A	0.999	0.841	0.835	0.809	0.999	0.810	0.812	0.766	0.826	0.824	0.788
	Gemini-2.5-Flash	N/A	0.913	0.732	0.747	0.684	0.868	0.635	0.627	0.609	0.684	0.687	0.647
	Qwen3.5-Plus	N/A	0.997	0.904	0.891	0.834	0.993	0.881	0.874	0.853	0.893	0.883	0.844
	GPT-5	N/A	0.998	0.912	0.921	0.892	0.995	0.884	0.887	0.848	0.898	0.904	0.870
Open-weights	DeepSeek-V4-Flash	284B	0.999	0.820	0.819	0.786	0.985	0.729	0.714	0.687	0.775	0.767	0.737

Table 1: Performance comparison between SocialBuddy and baseline models on the Easy and Hard subsets of the SocialSearch benchmark. (Note: Pre. denotes Precision.)

Figure 3: Data composition of personas, social relations, post types, and topics in SocialEnv.

Method / Variant	Prec.	Recall	EM
Qwen3.5-9B (Base)	0.783	0.778	0.732
+ SFT	0.865	0.868	0.835
+ OPD	0.921	0.924	0.909
+ RL (w/ F1-Score Reward only)	0.892	0.905	0.878
+ RL (w/ Macro Composite Reward)	0.914	0.921	0.894
+ RL (w/ Macro Reward+OPD)	0.941	0.943	0.921

Table 2: Ablation Study on Key Components of SocialPO.

Figure 4: Query intent characteristics in the SocialSearch benchmark.

Figure 5: Case study comparing Qwen3.5-397B-A17B and SocialBuddy-35B under complex multi-constraint social search.

Model	HR (%) $\downarrow$	Top-1 (%) $\uparrow$	Top-2 (%) $\uparrow$
Qwen3.5-35B-A3B	1.6	8.3	23.8
GLM-5.2	0.8	13.1	38.2
Qwen3.5-397B-A17B	1.3	21.4	46.5
SocialBuddy-35B (Ours)	1.4	64.2	91.5

Table 3: User experience evaluation results across

1{,}000

comparative trials in real-world personal social environments. HR, Top-1, and Top-2 denote Hallucination Rate, Top-1 Preference Rate, and Top-2 Preference Rate, respectively.

Tool	Operator	Parameters	Responsibility and evidence boundary
select_by_person	$\Omega_{\mathrm{author}}$	name ; relationship_type ; closeness ; gender ; age_group	Conjunctively filters identity, authorized social-graph attributes, and demographic cohorts. It does not infer authorship from post content or residence.
select_by_time	$\Omega_{\mathrm{temporal}}$	start_date ; end_date ; weekday ; festival ; time_of_day	Grounds absolute, periodic, and event-driven constraints in publication metadata rather than textual mentions; date bounds are inclusive.
select_by_location	$\Omega_{\mathrm{spatial}}$	keyword	Restricts spatial provenance to normalized poi_city or poi_name evidence attached to a post, explicitly excluding author residence.
select_by_content	$\Omega_{\mathrm{content}}$	post_type ; semantic_query	Applies an exact modality gate, then batched semantic assessment of principal-subject relevance over the survivors; private author fields and generated topics are withheld.
finalize	$R_{\rho}$	sort_by	Orders only eligible candidates by the requested objective and returns at most five sanitized records; it neither introduces a new relevance predicate nor re-filters the pool.

Table 4: Public action space of SocialBuddy. Each tool realizes a formal operator with an explicit parameterization and evidence boundary. Selection tools intersect their predicates with the active candidate pool; finalize is called exactly once at the end.

Argument	Type / status	Accepted input	Matching rule
Author metadata selection — select_by_person
name	string; optional	Free-form name or alias	Remove all whitespace and lowercase; retain an author when the query is a substring of either nickname or remark_name .
relationship_ type	enum; optional	self , family , relative , significant_other , colleague , college_classmate , school_classmate , friend , online_friend , acquaintance , neighbor , business_partner	Case-normalized exact match on the stored relationship category.
closeness	enum; optional	close , normal , distant , self	Case-normalized exact match on tie strength.
gender	enum; optional	male , female	Case-normalized exact match on profile metadata.
age_group	enum; optional	child , young , middle , senior	Maps respectively to stored categories child ; adolescent / young_adult ; adult / middle_aged ; and senior . Public interpretations are 7–12, 13–29, 30–65, and 66+.

Table 5: Implementation contract of select_by_person . Supplied arguments are conjoined, except that name may match either the normalized nickname or remark name.

Argument	Type / status	Accepted input	Matching rule
Publication-time selection — select_by_time
start_date	string; optional	YYYY-MM-DD	Inclusive lower calendar-date bound on post_time .
end_date	string; optional	YYYY-MM-DD	Inclusive upper calendar-date bound on post_time .
weekday	string; optional	Full name/abbreviation; 0 – 6 (Monday= 0 ); weekday ; weekend	Normalize to English weekday names and exactly match the recorded weekday field.
festival	enum; optional	One canonical festival label	Intersect with the recorded festival list. Spring Festival additionally includes its first-, second-, and third-day labels; no post-text inference is used.
time_of_day	enum; optional	morning , noon , afternoon , evening , night , midnight	Match half-open hour buckets $[05,11)$ , $[11,13)$ , $[13,17)$ , $[17,21)$ , $[21,24)$ , and $[00,05)$ , respectively.

Table 6: Implementation contract of select_by_time . Supplied predicates are conjoined over publication metadata; relative language is resolved by the policy before invocation.

Tool / argument	Type / status	Accepted input	Implementation and output
Spatial selection — select_by_location
keyword	string; required	City or POI phrase	Normalize case, apostrophes, and common trailing administrative suffixes; substring-match poi_city or poi_name . Update pool membership and report before/after counts.
Hybrid content selection — select_by_content
post_type	enum; optional	image , text , video , share_music , share_article	Apply an exact, in-memory modality gate before semantic inference.
semantic_query	string; optional	Open-vocabulary natural-language criterion	Select principal-subject matches from numbered post summaries, at most 100 per batch. Author fields, identifiers, raw time, weekday, and generated topics are withheld.
Terminal presentation — finalize

Table 7: Implementation contracts of the spatial, content, and terminal tools. Gray bands distinguish low-cost structured selection, hybrid content selection, and terminal presentation.

Role	Content
User	Query: show me HongJunYan’s text-only posts about reflecting on farming from Burqin in the evening over the last 30 days, sorted by most likes Reference time: Monday, September 8, 2025, 02:23.
Agent $\cdot$ Step 1	The query names HongJunYan, so I begin with the person filter. select_by_person(name="HongJunYan")
Observation	2,041 $\rightarrow$ 23. The normalized name matched remark_name=HongJunYan .
Agent $\cdot$ Step 2	Resolve “last 30 days” to August 10 through September 8 and retain evening posts. select_by_time(start_date="2025-08-10", end_date="2025-09-08", time_of_day="evening")
Observation	23 $\rightarrow$ 4. Matched the inclusive date range and evening bucket.
Agent $\cdot$ Step 3	Enforce the explicit Burqin provenance before inspecting content. select_by_location(keyword="Burqin")

Table 8: Successful trajectory combining named-author, rolling-window, time-of-day, location, text-modality, semantic, and likes constraints (task c559_0010 ).

Role	Content
User	Query: show me the video from friends last week late at night tagged in Weifang about showing off denim jackets with the most likes Reference time: Friday, July 4, 2025, 12:27.
Agent $\cdot$ Step 1	Restrict the pool to posts from friends. select_by_person(relationship_type="friend")
Observation	1,030 $\rightarrow$ 251. Relationship metadata matched.
Agent $\cdot$ Step 2	Resolve last week to June 23–29 and map late night to midnight. select_by_time(start_date="2025-06-23", end_date="2025-06-29", time_of_day="midnight")
Observation	251 $\rightarrow$ 11. The date range and midnight bucket matched.
Agent $\cdot$ Step 3	Apply the Weifang provenance constraint. select_by_location(keyword="Weifang")

Table 9: Successful trajectory combining friend scope, a resolved weekly interval, midnight publication time, video modality, semantic content, and likes ordering (task c94_0063 ).

Role	Content
User	Query: Find my posts from Spring Festival about grandpa saying goodnight at Gongheyong with the most comments. Reference time: Wednesday, March 4, 2026, 04:33.
Agent $\cdot$ Step 1	“My posts” maps to self-authorship. select_by_person(relationship_type="self")
Observation	8,663 $\rightarrow$ 23. The self relationship matched.
Agent $\cdot$ Step 2	Use the canonical Spring Festival annotation rather than a guessed date range. select_by_time(festival="Spring Festival")
Observation	23 $\rightarrow$ 2. Spring Festival and its first three day labels were included.
Agent $\cdot$ Step 3	Apply the Gongheyong POI constraint. select_by_location(keyword="Gongheyong")

Table 10: Successful trajectory grounding self-authorship, Spring Festival metadata, POI provenance, semantic content, and comments ordering (task c304_0003 ).

Role	Content
User	Query: posts by ShiRongPo about morning walk to school in Qingdao in the morning with the most comments Reference time: Wednesday, March 11, 2026, 03:54.
Agent $\cdot$ Step 1	The query names ShiRongPo. select_by_person(name="ShiRongPo")
Observation	7,003 $\rightarrow$ 52. The normalized name matched remark_name=ShiRongPo .
Agent $\cdot$ Step 2	The publication-time constraint is morning. select_by_time(time_of_day="morning")
Observation	52 $\rightarrow$ 14. Morning posts were retained.
Agent $\cdot$ Step 3	Apply the Qingdao provenance constraint. select_by_location(keyword="Qingdao")

Table 11: Successful multi-result trajectory in which comments ordering is observable after all four selectors retain two eligible posts (task c579_0053 ).

Role	Content
User	Query: friends posts about recording sunset over Kaiyuan on Saturday evening Reference time: Wednesday, April 22, 2026, 10:06.
Agent $\cdot$ Step 1	Restrict the pool to posts from friends. select_by_person(relationship_type="friend")
Observation	6,829 $\rightarrow$ 2,444. Relationship metadata matched.
Agent $\cdot$ Step 2	Retain posts published on Saturday evenings. select_by_time(weekday="Saturday", time_of_day="evening")
Observation	2,444 $\rightarrow$ 58. Weekday and evening predicates matched.
Agent $\cdot$ Step 3	Interpret Kaiyuan as an explicit post-location constraint. select_by_location(keyword="Kaiyuan")

Table 12: Diagnostic trajectory illustrating over-decomposition of a joint content phrase into an additional location constraint (task c504_0015 ).

Role	Content
User	Query: show posts about a child saying goodnight tagged at Kashgar posted late at night with the most likes Reference time: Thursday, October 23, 2025, 08:38.
Agent $\cdot$ Step 1	Interpret “a child” as an author-age constraint. select_by_person(age_group="child")
Observation	5,681 $\rightarrow$ 357. Child author profiles were retained.
Agent $\cdot$ Step 2	Map late night to the midnight bucket. select_by_time(time_of_day="midnight")
Observation	357 $\rightarrow$ 56. Midnight posts were retained.
Agent $\cdot$ Step 3	Apply the Kashgar location tag. select_by_location(keyword="Kashgar")

Table 13: Diagnostic trajectory illustrating confusion between a child depicted in a post and the post author’s age group (task c574_0006 ).

Facet	Source field(s)	Instantiated value
Demographic and setting
Identity	age_info , gender	34-year-old adult woman.
Location	location	Rural Guizhou, China; the source places her in the Qiandongnan Miao and Dong Autonomous Prefecture.
Education	education	Ph.D.; key-university background.
Household and work
Family	family	Married; lives with her spouse, two children, and father-in-law in a rural multigenerational household.

Table 16: Structured persona representation for GreenThumbChef, covering demographic context, education, occupation, household structure, daily routine, values, interests, and expression controls.

Facet	Source field(s)	Instantiated value
Demographic and setting
Identity	age_info , gender	81-year-old senior woman.
Location	location	Chongqing, China.
Education	education	Bachelor’s education; public-school background.
Household and work
Family	family	Widowed; two children; lives with her son’s family, including an adult grandson, in a multigenerational household.

Table 17: Structured persona representation for Grandma Li, characterized by late-life community participation, intergenerational care, and reflective expression.

Facet	Source field(s)	Instantiated value
Demographic and setting
Identity	age_info , gender	54-year-old middle-aged woman.
Location	location	Weifang, Shandong, China.
Education	education	College education; public-school background.
Household and work
Family	family	Married; lives with her spouse, adult son, and young granddaughter in a multigenerational household.

Table 18: Structured persona representation for Steady Hearth, coupling multigenerational family routines with responsible media supervision.

Facet	Source field(s)	Instantiated value
Demographic and setting
Identity	age_info , gender	51-year-old middle-aged woman.
Location	location	Chongqing, China.
Education	education	Bachelor’s education; private-school background.
Household and work
Family	family	Married; lives with her spouse, adult daughter, and elderly mother in a multigenerational urban household.

Table 19: Structured persona representation for Harmony Home, connecting ethical healthcare communication with multigenerational care.

Facet	Source field(s)	Instantiated value
Demographic and setting
Identity	age_info , gender	32-year-old adult woman.
Location	location	Coastal Guangdong, China.
Education	education	College education; private-school background.
Household and work
Family	family	Married; lives with her spouse, young daughter, and mother-in-law in a multigenerational urban household.

Table 20: Structured persona representation for Practical Bloom, organized around urban work logistics, child care, gardening, and cooking.

Field group	Included attributes	Role in synthesis	Exposure to social search
Public author layer
Author attributes	Age or age group, gender, surface identity, relationship type, and closeness.	Conditions author identity and voice.	Authorized categorical subset consumed by select_by_person .
Private generation context
Background context	Education, occupation, coarse home region, household composition, housing, and mobility.	Establishes persistent lived context.	Not directly exposed.
Behavioral context	Daily routine, values, life attitude, coping mechanism, interests, and narrative anchors.	Governs recurring activities, concerns, and interpretations.	Not directly exposed.
Expression controls	Screen name, posting archetype, tone, frequency, common-topic priors, and style description.	Regulates topic sampling and linguistic realization.	Not directly exposed as selector predicates.

Table 21: Field composition and visibility boundary of a SocialEnv persona record. Public author attributes support person-level retrieval, whereas private context conditions post synthesis but is not directly exposed to the search agent.

Type / ID	Time, context, engagement	Authored text and payload	Persona realization
A1 — Gardening milestone
text 005	2025-08-04 15:48:47 Monday; festival: none Topic: gardening POI: none 30 likes; 7 comments	Post: “The first tomato finally ripened on the vine this afternoon. A small, red milestone after months of tending.” Payload: Media/share: none.	Gardening is explicit; the measured “milestone” and restrained two-sentence caption instantiate patient maintenance and steady progress.
A2 — Rural commute pause
image 012	2025-08-27 17:23:00 Wednesday; festival: none Topic: cloud notes POI city: Long’e County POI name: withheld 40 likes; 5 comments	Post: “5:23 PM. The clouds over Long’e are putting on a show again. Just stopped the e-bike to look up before heading home to the kids.” Payload: 9 images; video count: none. Descriptions cover sunset clouds, an e-bike on a rural road, weather display, karst mountains, a wedding band, power lines, a puddle reflection, cloud detail, and fields at the horizon.	The rural commute, e-bike, return to her children, and deliberate pause jointly expose daily routine and the stated reflective coping pattern.
A3 — Dawn household routine
video 010	2025-08-16 05:30:55 Saturday; festival: none Topic: home cooking POI: none 12 likes; 0 comments	Post: “5:30 AM. The kitchen is quiet while everyone sleeps. Starting the congee before the morning rush begins.” Payload: Image count: none; 1 video describing rice congee simmering on a gas stove in a dim kitchen, with steam rising from the pot.	Early meal preparation materializes planning and care for a multigenerational household; the precise timestamp and wording remain methodical.

Table 22: Cross-modal post records for Persona A (GreenThumbChef). “Payload” reports each applicable modality-specific field; none marks nullable fields, and long media descriptions or supplements are summarized.

Type / ID	Time, context, engagement	Authored text and payload	Persona realization
B1 — Traditional family lunch
text 012	2025-09-29 12:21:45 Monday; festival: none Topic: Cooking traditional Chinese cuisine POI: none 44 likes; 2 comments	Post: “Lunch is ready: braised pork and steamed fish. The grandson loved it. A quiet Monday at home.” Payload: Media/share: none.	Traditional cooking and care for her adult grandson appear in a concise domestic update consistent with a gentle Family Elder voice.
B2 — Reciprocal digital learning
image 042	2026-05-08 17:12:36 Friday; festival: none Topic: iPad POI: none 26 likes; 7 comments	Post: “Late afternoon light filters through the window, warming the screen where my grandson teaches me to paint. Time slows down in these quiet, colorful moments.” Payload: 9 images; video count: none. Descriptions show a tablet, stylus and watercolor interface, an elderly learner and younger guide, plum-blossom painting, tea, warm window light, a sketch gallery, and the pair seated together.	Reciprocal learning reverses the usual direction of care while preserving intergenerational continuity and openness to new practices.
B3 — Tai Chi routine
video 002	2025-07-04 19:08:25 Friday; festival: none Topic: Tai Chi POI: none 30 likes; 8 comments	Post: “Even at 81, my Tai Chi flows strong in the summer evening breeze! So proud of keeping active for my family.” Payload: Image count: none; 1 video describing an elderly woman performing slow Tai Chi movements in a community park at dusk.	Age, established exercise routine, community setting, and family-oriented motivation are simultaneously observable.

Table 23: Cross-modal post records for Persona B (Grandma Li).

Type / ID	Time, context, engagement	Authored text and payload	Persona realization
C1 — Work–family continuity
text 021	2025-12-16 14:12:57 Tuesday; festival: none Topic: seasonal greetings POI city: Weifang POI name: media workplace (coarsened) 31 likes; 10 comments	Post: “Just finished a quick lunch at the office. The winter sun is weak today, but seeing my granddaughter’s drawing of snowflakes on my desk warms my heart. Can’t wait to pick her up and make hot pear soup for everyone tonight. Winter in Mihe is cold, but our home is cozy.” Payload: Media/share: none.	Office work, kindergarten pickup, family cooking, season, and a warm domestic register are integrated in one record.
C2 — Intergenerational overnight care
image 003	2025-06-06 03:14:46 Friday; festival: none Topic: diary POI: none 22 likes; 3 comments	Post: “3:14 AM and the house is finally quiet. My granddaughter had a fever tonight, so her mom and I took turns watching her. She’s sleeping soundly now. I’ll brew some ginger tea for myself before heading back to bed. These late nights remind me how fast time flies; my son was once the one I stayed up with, and now I’m doing it for his little girl. Summer nights are warm, but the worry keeps you alert. Goodnight, world.” Payload: 1 image; video count: none. The description shows a dim living room, amber tea, a digital clock reading 03:14, and a summer blanket.	Overnight care links past parenting to present grandparenting and makes multigenerational cooperation observable.
C3 — Professional reflection
video 006	2025-07-08 01:59:29 Tuesday; festival: none Topic: work POI: none 0 likes; 0 comments	Post: “It’s late, but the office is finally quiet. After decades in media, I’ve learned that the best decisions often come not in the rush of the day, but in these still moments. Watching the city sleep from my window, everything feels clear and in its place.” Payload: Image count: none; 1 video describing a dim office, desk lamp, papers, dark monitor, rain-streaked window, and city lights.	“Decades in media” grounds the occupation, while deliberate judgment and quiet reflection align with steady, responsible leadership.

Table 24: Cross-modal post records for Persona C (Steady Hearth).

Explore similar work

May 27, 2026

VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild

LLM-based agents score well on search benchmarks, yet real users consistently find results unsatisfying, revealing a persistent evaluation-experience gap. We attribute this gap to existing benchmarks' reliance on over-specified queries, single-turn interactions, and fixed-schema evaluation, none of which reflect real search behavior where users and agents collaboratively refine vague intent through multi-turn dialogue. We term this paradigm VibeSearch and introduce VibeSearchBench, a benchmark comprising 200 manually curated bilingual (Chinese and English) tasks across 20 domains, split into VibeSearch-Pro (professional) and VibeSearch-Daily (daily-life) subsets. Each task pairs a user persona with a schema-free ground-truth knowledge graph, and is evaluated through a progressive-disclosure user simulator and a graph-matching evaluation framework. We benchmark seven frontier models under both the ReAct framework and the OpenClaw agent harness. Results show that all models remain substantially inadequate for VibeSearch (best F1: 30.30), highlighting the need for fundamental advances in long-context reasoning, proactive intent elicitation, and structured knowledge construction.

Xiaohongshu Inc

Xiaohongshu Dots Studio & Unipat AI

Agentic Search Long-Horizon Agent Evaluation Conversational Search AI Agent Benchmarks

Ask PDF

Jul 25, 2026

SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task

Recent advances in large language models (LLMs) have enabled search agents to autonomously tackle complex tasks across extended search and reasoning horizons. However, training effective search agents remains challenging due to the lack of scalable and long-horizon tasks, and the difficulty of evaluating and correcting intermediate reasoning and tool-use behaviors. We introduce SearchArt, a scalable framework for training long-horizon search agents through verification-driven task synthesis and a multi-stage post-training pipeline. SearchArt constructs large-scale datasets for complex search-, research- and user-oriented tasks by synthesizing diverse information-seeking QA pairs and corresponding search trajectories from web documents and automatically generated evidence graphs. To ensure the reliability of the synthesized data, we design a verification pipeline that jointly evaluates QA consistency, trajectory quality, and the relevance of retrieved evidence. The verified trajectories are subsequently used in a multi-stage training process comprising supervised fine-tuning and reinforcement learning-based policy optimization. Search agents trained with SearchArt exhibit adaptive search planning, iterative evidence aggregation, and complex reasoning over extended interaction horizons. Experimental results demonstrate that, with only (Qwen3.5-) 27B parameters, SearchArt scores 74.39 on BrowseComp-ZH, 70.06 on BrowseComp, and 52.55 on Deepresearch-bench, matching or surpassing frontier closed-source agents on both deepsearch and deepresearch benchmarks.

Lang Mei, Xiaohan Yu, Chong Chen +27

Huawei Cloud Post-Training Team

Synthetic Data Generation LLM Agents LLM Agent Training Long-Horizon LLM Agents Autonomous Research Agents

Ask PDF

Apr 17, 2026

SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems

As Large Language Models (LLMs) transition from text processors to autonomous agents, evaluating their social reasoning in embodied multi-agent settings becomes critical. We introduce SocialGrid, an embodied multi-agent environment inspired by Among Us that evaluates LLM agents on planning, task execution, and social reasoning. Our evaluations reveal that even the strongest open model (GPT-OSS-120B) achieves below 60% accuracy in task completion and planning, with agents getting stuck in repetitive behaviors or failing to navigate basic obstacles. Since poor navigation confounds evaluation of social intelligence, SocialGrid offers an optional Planning Oracle to isolate social reasoning from planning deficits. While planning assistance improves task completion, social reasoning remains a bottleneck: agents fail to detect deception at near-random chance regardless of scale, relying on shallow heuristics rather than accumulating behavioral evidence. SocialGrid provides automatic failure analysis and fine-grained metrics, enabling developers to diagnose and improve their agents. We also establish a competitive leaderboard using Elo ratings from adversarial league play.

Hikaru Shindo, Hanzhao Lin, Lukas Helff +2

Technical University of Darmstadt · German Research Center for Artificial Intelligence · CERTAIN +2

LLM Agent Evaluation AI Agent Benchmarks Multi-Agent Systems Multi-Agent Reasoning LLM Agent Planning

Ask PDF