OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions

Fangzhi Xu; Hang Yan; Qiushi Sun; Jinyang Wu; Zixian Huang; Muye Huang; Jingyang Gong; Zichen Ding

cs.CLFeb 5, 2026

OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions

Authors: Fangzhi Xu, Hang Yan, Qiushi Sun, Jinyang Wu, Zixian Huang, Muye Huang, Jingyang Gong, Zichen Ding,

Organizations: Xi’an Jiaotong University · The University of Hong Kong · Tsinghua University · Shanghai AI Laboratory ·

Abstract

The advancement of Large Language Models (LLMs) has catalyzed the development of autonomous agents capable of navigating complex environments. However, existing evaluations primarily adopt a deductive paradigm, where agents execute tasks based on explicitly provided rules and static goals, often within limited planning horizons. Crucially, this neglects the inductive necessity for agents to discover latent transition laws from experience autonomously, which is the cornerstone for enabling agentic foresight and sustaining strategic coherence. To bridge this gap, we introduce OdysseyArena, which re-centers agent evaluation on long-horizon, active, and inductive interactions. We formalize and instantiate four primitives, translating abstract transition dynamics into concrete interactive environments. Building upon this, we establish OdysseyArena-Lite for standardized benchmarking, providing a set of 120 tasks to measure an agent's inductive efficiency and long-horizon discovery. Pushing further, we introduce OdysseyArena-Challenge to stress-test agent stability across extreme interaction horizons (e.g., > 200 steps). Extensive experiments on 15+ leading LLMs reveal that even frontier models exhibit a deficiency in inductive scenarios, identifying a critical bottleneck in the pursuit of autonomous discovery in complex environments. Project homepage: https://os-copilot.github.io/OdysseyArena-Home/

Figures & tables

Figure 1: Comparison between deductive and inductive settings in multi-turn agentic tasks.

Benchmark	Ind.	Horizon	Deploy
BabyAI [ 2019 ]	✓	Long	Simulator
ALFWorld [ 2021 ]	✓	Short	Simulator
GAIA [ 2023 ]	✗	Long	Offline
WebArena [ 2024 ]	✗	Short	Docker
OSWorld [ 2024 ]	✗	Long	Docker
AndroidWorld [ 2025 ]	✗	Short	Emulator

Table 1: Comparison of representative multi-turn agentic benchmarks. Ind. indicates whether inductive reasoning is required. Horizon denotes the number of steps required to complete a task, categorized as short (

<

50), long (50–100), or X-Long (

>

100). Deploy describes the evaluation setup required to run the environment, with API-based deployment being the most lightweight.

Figure 2: Demonstrations of four OdysseyArena environments: Turn On Lights, AI Trading, Energy Dispatch, and Repo System. For clarity, we omit the task prompts here and present only the interaction trajectories. Full prompts are provided in Appendix C .

Figure 3: Overview of the benchmark architecture, illustrating the environment configuration initialization (left) and the interaction loop between the LLM agent and the environment step logic (right).

Model	Turn On Lights		AI Trading		Energy Dispatch		Repo System
Model	Avg@4	Pass@4	Avg@4	Best@4	Avg@4	Pass@4	Avg@4	Pass@4
Human	81.67	100.00	+92.55%	+197.23%	25.00	60.00	77.50	100.00
Gemini 3 Pro Preview	44.17	76.67	+67.71%	+76.94%	30.00	36.67	65.83	80.00
GPT-5	28.33	40.00	+17.32%	+20.47%	23.33	40.00	62.50	83.33
Gemini 2.5 Pro	29.17	50.00	+33.02%	+40.12%	10.83	26.67	50.00	66.67
gpt-oss-120b (high)	27.50	40.00	+23.27%	+27.47%	0.00	0.00	18.33	33.33

Table 2: Performance comparison on four environments. We provide three different reasoning effort of gpt-oss-120b. For AI Trading environment, we report the profit rate and Best@4 is calculated based on the highest profit of each task. For other three environments, we report the success rate. Colored Rows represent proprietary models. The best results are in bold .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Action	Description
Turn On Lights	Toggle <light>	The parameter light is the specific index of the light you want to toggle. This action is successfully executed only if its latent logical condition is satisfied.
AI Trading	{ "Buy":<stocks, shares>, "Sell":<stocks, shares> }	Purchase specific shares of stock and sell specific shares of stock. Multiple stocks can be acquired or sold simultaneously. The buy action is successful only if the cash is enough. If the selling shares exceed the current holdings, the environment defaults to selling the entire existing position.
Energy Dispatch	{ "Thermal":<p 1 >, "Wind":<p 2 >, "Solar":<p 3 >, "Battery":<p 4 > }	Dispatch p 1 , p 2 , p 3 units of thermal, wind, and solar power respectively. For p 4 , a negative value represents charging and positive represents discharging. Notably, charging is truncated upon reaching the battery’s max capacity, and the discharging is limited to the available capacity if it exceeds the current state of charge. The cost of the four energies should be limited within the total budget.
Repo System	repo tree/ls	Inspect the repository’s directory structure to identify script paths.
	pip install <package>	Install the package . Supports specific version ( == ) and range constraints ( $>$ , $>=$ , $<$ , $<=$ ).
	pip uninstall <package>	Remove the specified package from the configuration.

Appendix

Table 4: Action Space of and description for OdysseyArena . All parameters are wrapped in <> .

Model	Turn On Lights		AI Trading		Energy Dispatch		Repo Management
Model	w/o rules	w/ rules	w/o rules	w/ rules	w/o rules	w/ rules	w/o rules	w/ rules
Gemini 3 Pro Preview	44.17	100.00	+67.71%	+135.48%	30.00	16.67	65.83	96.67
GPT-5	28.33	100.00	+17.32%	+132.02%	23.33	13.33	62.50	96.67
Gemini 2.5 Pro	29.17	100.00	+33.02%	+126.24%	10.83	20.00	50.00	96.67
gpt-oss-120b (high)	27.50	100.00	+23.27%	+141.63%	0.00	0.00	18.33	50.00
DeepSeek-V3.2	18.33	96.67	+8.62%	+118.88%	0.00	0.00	48.33	56.67

Appendix

Table 5: Performance comparison between w/o rules and w/ rules settings. We provide three different reasoning effort of gpt-oss-120b. For AI Trading environment, we report the profit rate and pass@4 is calculated based on the highest profit of each task. For other three environments, we report the success rate. Colored Rows represent proprietary models.

Figure 7: All data for Success Rate against Step. We do not plot for Energy Dispatch environment due to its complex success conditions.

Figure 8: Step density distribution for LLM models: (a) Gemini 3 Pro Preview, (b) GPT-5, (c) Gemini 2.5 Pro, (d) gpt-oss-120b (high), (e) DeepSeek-V3.2, (f) Grok 4 Fast, (g) Qwen3-235B-A22B-Instruct, (h) gpt-oss-120b (medium), (i) Qwen3-30B-A3B-Instruct, (j) GLM-4-32B-0414, (k) gpt-oss-120b (low), (l) Llama 3.3 70B Instruct, (m) Qwen3-4B-Instruct (n) Llama 3.1 8B Instruct, (o) GLM-4-9B-Chat.

Figure 9: Task success status (based on pass@4 ). Each row represents: (a) Human, (b) Gemini3 Pro Preview, (c) GPT-5, (d) Gemini 2.5 Pro, (e) gpt-oss-120b (high), (f) DeepSeek-V3.2, (g) Grok 4 Fast, (h) Qwen3-235B-A22B-Instruct, (i) gpt-oss-120b (medium), (j) Qwen3-30B-A3B-Instruct, (k) GLM-4-32B-0414, (l) gpt-oss-120b (low), (m) Llama 3.3 70B Instruct, (n) Qwen3-4B-Instruct, (o) Llama 3.1 8B Instruct, (p) GLM-4-9B-Chat. Dark green cells indicate tasks solved by Human. Green cells indicate tasks solved by LLM agents. Gray cells indicate unsolved tasks. We report the average success rate in Repo System across all LLMs for each subset (Easy, Medium and Hard).

Model	Avg@4				Pass@4
Model	200 turns	150 turns	100 turns	50 turns	200 turns	150 turns	100 turns	50 turns
Gemini 3 Pro Preview	65.83	70.00	76.67	70.83	80.00	83.33	86.67	83.33
gpt-oss-120b-high	18.33	19.17	23.33	11.67	33.33	33.33	36.67	20.00

Appendix

Table 6: Performance in Repo System with different memory usage.

Model	Original Avg@4	In-Context Avg@4	Original Pass@4	In-Context Pass@4
Gemini 3 Pro Preview	65.83	66.67	80.00	81.67
gpt-oss-120b-high	18.33	17.50	33.33	30.00

Appendix

Table 7: In-Context Learning Performance Comparison in Repo System

Figure 10: The line chart represents token usage and the bar chart represents token efficiency. Token Usage is measured in units of

10^{6}

, while Token Efficiency is reported in units of

10^{-6}

Table 16

Figure 11: Screenshot of the user interface and instructions provided to the human annotators.

Environment	Success Rate Type	Metric	Inner-Annotation Score
Turn On Lights	Discrete	Fleiss’ $\kappa$	0.42
Energy Dispatch	Discrete	Fleiss’ $\kappa$	0.40
AI Trading	Continuous	Intraclass Correlation Coefficient	0.12
Repo System	Discrete	Fleiss’ $\kappa$	0.18

Appendix

Table 10: Inter-rater reliability analysis across four experimental environments. For tasks with discrete success metrics ( Turn On Lights , Energy Dispatch , and Repo System ), we report Fleiss’ Kappa (

\kappa

). For the continuous reasoning task ( AI Trading ), the Intraclass Correlation Coefficient (ICC) is employed.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks
Feb 18, 2026Tanqiu Jiang, Yuhui Wang, Jiacheng Liang +1LLM Agent Security AI Agent Security Benchmarks
Department of Computer Science, Stony Brook University, Stony Brook, USA.
Escaping the Context Bottleneck: Active Context Curation for LLM Agents via Reinforcement Learning
Apr 13, 2026Xiaozhe Li, Tianyi Lyu, Yizhao Yang +6Lost-in-the-Middle Effect LLM Context Management
Tongji University · CurrentsAI Research · Stanford University +1
AdaExploreBench: Benchmarking LLM Agents for Proactive Exploration and Test-time Adaptation in Evolving Environments
Oct 19, 2026Rui Wang, Zishen Lai, Jiayu Liu +15LLM Agent Evaluation AI Agent Benchmarks
HKUST · HKUST-UGAIL · HKUST(GZ) +3

Feb 18, 2026J/K move · Enter open · S save

AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks

Tanqiu Jiang, Yuhui Wang, Jiacheng Liang, Ting Wang

Department of Computer Science, Stony Brook University, Stony Brook, USA.

LLM Agent Security AI Agent Security Benchmarks Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

LLM agents are increasingly deployed in long-horizon, complex environments to solve challenging problems, but this expansion exposes them to long-horizon attacks that exploit multi-turn user-agent-environment interactions to achieve objectives infeasible in single-turn settings. To measure agent vulnerabilities to such risks, we present AgentLAB, the first benchmark dedicated to evaluating LLM agent susceptibility to adaptive, long-horizon attacks. Currently, AgentLAB supports five novel attack types including intent hijacking, tool chaining, task injection, objective drifting, and memory poisoning, spanning 28 realistic agentic environments, and 644 security test cases. Leveraging AgentLAB, we evaluate representative LLM agents and find that they remain highly susceptible to long-horizon attacks; moreover, defenses designed for single-turn interactions fail to reliably mitigate long-horizon threats. We anticipate that AgentLAB will serve as a valuable benchmark for tracking progress on securing LLM agents in practical settings. The benchmark is publicly available at https://tanqiujiang.github.io/AgentLAB_main.

Abstract

AI Agent Benchmarks

Explore similar work

AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks

Escaping the Context Bottleneck: Active Context Curation for LLM Agents via Reinforcement Learning

AdaExploreBench: Benchmarking LLM Agents for Proactive Exploration and Test-time Adaptation in Evolving Environments