OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence

Peigen Liu; Rui Ding; Yuren Mao; Ziyan Jiang; Yuxiang Ye; Yunjun Gao; Ying Zhang; Renjie Sun

cs.AIMar 16, 2026

OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence

Authors: Peigen Liu, Rui Ding, Yuren Mao, Ziyan Jiang, Yuxiang Ye, Yunjun Gao, Ying Zhang, Renjie Sun,

Organizations: School of Software Technology, Zhejiang University · Laboratory for Statistical Monitoring and Intelligent Governance of Common Prosperity, Zhejiang Gongshang University · Tongyi Lab, Alibaba Group · Alibaba Cloud

Abstract

Large Language Model (LLM)-based Collective Intelligence (CI) presents a promising approach to overcoming the data wall and continuously boosting the capabilities of LLM agents. However, there is currently no dedicated arena for evolving and benchmarking LLM-based CI. To address this gap, we introduce OpenHospital, an interactive arena where physician agents can evolve CI through interactions with patient agents. This arena employs a data-in-agent-self paradigm that rapidly enhances agent capabilities and provides robust evaluation metrics for benchmarking both medical proficiency and system efficiency. Experiments demonstrate the effectiveness of OpenHospital in both fostering and quantifying CI.

Figures & tables

Metric	Score
Self-BLEU4	0.4111
TF-IDF Diversity	0.8727
Perplexity	5.7501
Medical Consistency	4.4113

Table 1: Evaluation results of the synthesized static patient data.

Figure 1: Interactional response diversity of patient agents across the fixed question set.

Figure 2: Dynamic response quality scores evaluated by GPT-5.2 across Accuracy, Relevance, and Persona Alignment.

Figure 3: Diagnostic evolution of a physician agent across training batches.

Figure 4: Performance trajectories of clinical capabilities across training batches.

Figure 5: Trend of Total Token Consumption across training batches.

Figure 6: An example of cross-departmental physician consultation.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 705 papers

ClinVerse: A Realistic Multi-Agent Diagnostic Environment for Training and Evaluating Medical Agents
Oct 26, 2026Sirry Chen, Fangke Chen, Xuanwen Ding +2Medical AI Agents LLM Agent Training
Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Zhejiang University, Hangzhou, China +2
PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
May 4, 2026Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler +10AI Agent Benchmarks Electronic Health Records
Stanford University
When OpenClaw Meets Hospital: Toward an Agentic Operating System for Dynamic Clinical Workflows
Mar 12, 2026Wenxian Yang, Hanzheng Qiu, Bangqun Zhang +5Workflow Automation LLM Agent Memory
Independent Researcher · National Institute for Data Science in Health and Medicine, Xiamen University, Xiamen, China · Yue’erwan Internet Hospital Co., Ltd. +4

Oct 26, 2026J/K move · Enter open · S save

ClinVerse: A Realistic Multi-Agent Diagnostic Environment for Training and Evaluating Medical Agents

Sirry Chen, Fangke Chen, Xuanwen Ding, Wei Chen+1

Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · Zhejiang University, Hangzhou, China+2

Medical AI Agents LLM Agent Training Medical Diagnosis LLM Agent Evaluation Multi-Agent LLMs

Open paper AskPDF arXiv

Disease diagnosis is a key application of large language model (LLM) agents in healthcare. However, existing medical agents are primarily trained and evaluated with static question-answering data, providing limited opportunities to learn active information acquisition, examination decision-making, and dynamic clinical interaction. To address this limitation, we present ClinVerse, an interactive, real-data-grounded multi-agent environment designed for training and evaluating medical diagnostic agents. Built on ClinPatient, a scalable dataset comprising 150k diagnostic cases derived from real-world electronic health records, ClinVerse supports coordinated interactions among six specialized clinical roles. Specifically, a doctor agent starts from an initial patient complaint, progressively gathers clinical evidence through patient interviews and examination requests, and ultimately produces a diagnosis. For each interaction trajectory, a rule-based scoring framework quantifies information acquisition, examination ordering, and diagnostic accuracy. These trajectory-level scores serve as reward signals for agent training while also enabling scalable evaluation. In experiments, we evaluate various general-purpose and medical-domain models and show that nine self-evolution methods across three categories consistently improve agents' diagnostic capabilities within ClinVerse. We further analyze the interplay among model capability, environment feedback, and self-evolution methods.

Abstract

LLM Agents

Explore similar work

ClinVerse: A Realistic Multi-Agent Diagnostic Environment for Training and Evaluating Medical Agents

PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments

When OpenClaw Meets Hospital: Toward an Agentic Operating System for Dynamic Clinical Workflows