cs.AIApr 13, 2026

Collaborative Multi-Agent Scripts Generation for Enhancing Imperfect-Information Reasoning in Murder Mystery Games

Authors: Keyang Zhong, Junlin Xie, Hefeng Wu, Haofeng Li, Guanbin Li

Organizations: Sun Yat-sen University · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-language models (VLMs) have shown impressive capabilities in perceptual tasks, yet they degrade in complex multi-hop reasoning under multiplayer game settings with imperfect and deceptive information. In this paper, we study a representative multiplayer task, Murder Mystery Games, which require inferring hidden truths based on partial clues provided by roles with different intentions. To address this challenge, we propose a collaborative multi-agent framework for evaluating and synthesizing high-quality, role-driven multiplayer game scripts, enabling fine-grained interaction patterns tailored to character identities (i.e., murderer vs. innocent). Our system generates rich multimodal contexts, including character backstories, visual and textual clues, and multi-hop reasoning chains, through coordinated agent interactions. We design a two-stage agent-monitored training strategy to enhance the reasoning ability of VLMs: (1) chain-of-thought based fine-tuning on curated and synthetic datasets that model uncertainty and deception; (2) GRPO-based reinforcement learning with agent-monitored reward shaping, encouraging the model to develop character-specific reasoning behaviors and effective multimodal multi-hop inference. Extensive experiments demonstrate that our method significantly boosts the performance of VLMs in narrative reasoning, hidden fact extraction, and deception-resilient understanding. Our contributions offer a scalable solution for training and evaluating VLMs under uncertain, adversarial, and socially complex conditions, laying the groundwork for future benchmarks in multimodal multi-hop reasoning under imperfect information.

Figures & tables

Figure 1: Overview of the proposed framework. It employs evaluation agents and generation agents to collaboratively generate logically coherent game scripts and instructs a pretrained VLM via a two-stage training strategy under agent monitoring to enhance the target model’s reasoning capability under imperfect information.
Figure 2: The details of game scripts generated via our multi-agent framework.
Figure 3: The bottom part outlines the two-stage training strategy. The top part showcases the ScoreAgent design, which applies specialized reward functions to different training data types for reward calculation during reinforcement learning.
Figure 4: Red denotes low-scoring and green denotes high-scoring examples. Low scores arise from off-topic, self-contradictory, or rule-violating behaviors in self-introductions and discussions, such as irrelevance to the script or premature identity revelation.
MethodReasoning & AnalysisRole-playing & DecisionPerception
MMRCMDRPDMLSUTIUMIU
Proprietary VLMs
GPT-4V58.7526.436.4324.2%92.4051.8869.25
Gemini-1.5-Pro57.3919.207.2216.9%88.8057.7857.84
Claude57.7822.077.8919.2%88.8035.3155.02
Open-source VLMs
Table 1: Performance comparison across seven metrics grouped by capability types. Bold denotes the best result and underline indicates the second-best result across all methods. ⋆ indicates that our model surpasses all open-source VLMs. Results are reported for both Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct backbones.
Figure 5: Training reward curves for verifiable subtasks (Multimodal and Long Scripts QA, top row) and unverifiable role-playing tasks (Self-Introduction and Discussion, bottom row).
DataMMRCMDRPDMLSUTIUMIU
Human51.0330.726.0432.21%79.0569.6156.42
Synthetic48.3527.905.4134.21%82.0268.4554.52
Human+Syn55.0134.256.3535.00%87.4074.5661.09
Table 2: Ablation on training data sources using Qwen2.5-VL-3B-Instruct with all evaluation sets are human-annotated.
Judge PairCMD rrRP rr
GPT-4o ↔\leftrightarrow DeepSeek-r10.830.68
Gemini-2.5-Pro ↔\leftrightarrow DeepSeek-r10.790.64
GPT-4o ↔\leftrightarrow Gemini-2.5-Pro0.870.72
Table 3: Pairwise Pearson correlation between different LLM judges ( p<10−2p<10^{-2} for all cases).
TaskSpearman ρ\rhopp -value
CMD0.71<10−2<10^{-2}
RP0.62<10−2<10^{-2}
Table 4: Correlation between human judgments and aggregated LLM-as-Judge scores.
Task-1 ∼\sim 01 ∼\sim 23 ∼\sim 4>> 4
RP (Human vs. Avg LLM)63%23%10%4%
RP (Human vs. DeepSeek)58%24%12%6%
Task-2 ∼\sim 01 ∼\sim 34 ∼\sim 6>> 6
CMD (Human vs. Avg LLM)48%32%15%5%
CMD (Human vs. DeepSeek)46%30%18%6%
Table 5: Distribution of score differences between human judgments and LLM-based evaluations.
Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Statistics of the proposed dataset: (a) Distribution of perception QA; (b) Distribution of the number of roles in the scripts; (c) Distribution of reasoning steps for cognition assessments.
Figure 7: An example of synthetic Multimodal Clues Pool
Figure 8: An example of synthetic Role Scripts
Figure 9: System prompt of OutlineAgent
Figure 10: System prompt of CharacterAgent
Figure 11: System prompt of CriticAgent
Figure 12: System prompt of ClueAgent which include both text clues and images clues prompt.
Figure 13: System prompt of QaAgent for multihot reasoning chain generation.
Figure 14: System prompt of QaAgent for long scripts question-answer pairs generation.
Figure 15: System prompt of QaAgent for one-hop image-based question-answer pairs generation, which includes both text-rich and media-rich questions.
Figure 16: System prompt of QaAgent for multi-hop multimodal question-answer pairs generation.
Figure 17: System prompt of RoleplayAgent for first-person self-introduction generation.
Figure 18: System prompt of RoleplayAgent for ask other player questions generation.
Figure 19: System prompt of RoleplayAgent for answer other players’ questions generation.
Figure 20: System prompt of RoleplayAgent for answer other players’ questions generation.
Figure 21: Prompt Templates of Judge LLM for scoring different type tasks’ response.
More in this topic

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Explore similar work

CardsList
Nov 14, 2025

Multi-agent Undercover Gaming: Hallucination Removal via Counterfactual Test for Multimodal Reasoning

Hallucination continues to pose a major obstacle in the reasoning capabilities of large language models (LLMs). Although the Multi-Agent Debate (MAD) paradigm offers a promising solution by promoting consensus among multiple agents to enhance reliability, it relies on the unrealistic assumption that all debaters are rational and reflective, which is a condition that may not hold when agents themselves are prone to hallucinations. To address this gap, we introduce the Multi-agent Undercover Gaming (MUG) protocol, inspired by social deduction games like "Who is Undercover?". MUG reframes MAD as a process of detecting "undercover" agents (those suffering from hallucinations) by employing multimodal counterfactual tests. Specifically, we modify reference images to introduce counterfactual evidence and observe whether agents can accurately identify these changes, providing ground-truth for identifying hallucinating agents and enabling robust, crowd-powered multimodal reasoning. MUG advances MAD protocols along three key dimensions: (1) enabling factual verification beyond statistical consensus through counterfactual testing; (2) introducing cross-evidence reasoning via dynamically modified evidence sources instead of relying on static inputs; and (3) fostering active reasoning, where agents engage in probing discussions rather than passively answering questions. Collectively, these innovations offer a more reliable and effective framework for multimodal reasoning in LLMs. The source code can be accessed at https://github.com/YongLD/MUG.git.

Dayong Liang, Xiao-Yong Wei, Linhui Xiao +1

School of Future Technology, South China University of Technology, Guangzhou, China · Department of networked intelligence, Peng Cheng Laboratory, Shenzhen, China · College of Computer Science, Sichuan University, Chengdu, China +1

Multi-Agent LLM SystemsMultimodal ReasoningMultimodal HallucinationHallucination Detection in VLMs
AskPDF
Aug 25, 2026

DoublesEval: Diagnosing Multi-Agent Tactical Reasoning in Vision-Language Models via Professional Doubles Badminton

Visual Language Models (VLMs) excel at describing visible scene content but struggle to reason about dynamic multi-agent interactions, where action semantics depend on coordinated roles and spatial-temporal dependencies. We formalize this capability as multi-agent tactical reasoning and introduce DoublesEval, a diagnostic evaluation framework that leverages professional doubles badminton as a structurally tractable testbed. DoublesEval employs a key-moment-based protocol that decomposes rallies into tactically salient instants and probes models across four interpretable dimensions: atomic recognition, intra-segment composite understanding, cross-segment causal reasoning, and high-level tactical abstraction. This design isolates where reasoning fails, rather than merely measuring answer correctness. To address observed failure modes, we propose TacticCheck, a lightweight constraint-guided test-time consistency checker that reranks candidate answers using the model's own lower-level tactical predictions, requiring no parameter updates or ground-truth labels at inference time. Evaluating four representative open-source VLMs on 60 curated rallies (yielding ∼\sim9.6K structured instances) via a zero-shot protocol, we find that models remain weak across all diagnostic levels, with especially clear bottlenecks in spatial state, interaction binding, and terminal evidence. TacticCheck delivers consistent gains across all evaluated models, while still leaving a substantial gap to robust tactical reasoning. These results highlight the need for structured, interaction-aware evaluation paradigms for next-generation VLMs. The source code is available in blue{our GitHub repository}.

Jintao Cheng, Weibin Li

The Hong Kong University of Science and Technology · University of Macau

VLM EvaluationVLM ReasoningVideo Reasoning
AskPDF
Mar 17, 2026

How Clued up are LLMs? Evaluating Multi-Step Deductive Reasoning in a Text-Based Game Environment

Deducing whodunit proves challenging for LLM agents. In this paper, we implement a text-based multi-agent version of the classic board game Clue as a rule-based testbed for evaluating multi-step deductive reasoning, with six agents drawn from GPT-4o-mini and Gemini-2.5-Flash. We further investigate whether fine-tuning on structured logic puzzles transfers to improved in-game reasoning and gameplay. Across 18 simulated games, agents achieve only four correct wins, indicating difficulty in maintaining consistent deductive reasoning over the course of a full game. Additionally, we find that fine-tuning does not reliably improve performance and, in some cases, appears to increase reasoning volume without improving reasoning precision.

Rebecca Ansell, Autumn Toney-Wails

Department of Computer Science, Georgetown University, Washington, DC, USA · Syntheos, Corp; UNU-Merit, Maastricht, NL

Fine-Tuning Language ModelsLLM Agent EvaluationFine-TuningMulti-Agent LLM SystemsGame-Playing Agents
AskPDF