OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions
Authors: Fangzhi Xu, Hang Yan, Qiushi Sun, Jinyang Wu, Zixian Huang, Muye Huang, Jingyang Gong, Zichen Ding, Kanzhi Cheng, Yian Wang, Xinyu Che, Zeyi Sun, Jian Zhang, Zhangyue Yin, Haoran Luo, Ben Kao, Qika Lin
Organizations: Xi’an Jiaotong University · The University of Hong Kong · Tsinghua University · Shanghai AI Laboratory · Nanjing University · National University of Singapore · Fudan University · Nanyang Technological University
The advancement of Large Language Models (LLMs) has catalyzed the development of autonomous agents capable of navigating complex environments. However, existing evaluations primarily adopt a deductive paradigm, where agents execute tasks based on explicitly provided rules and static goals, often within limited planning horizons. Crucially, this neglects the inductive necessity for agents to discover latent transition laws from experience autonomously, which is the cornerstone for enabling agentic foresight and sustaining strategic coherence. To bridge this gap, we introduce OdysseyArena, which re-centers agent evaluation on long-horizon, active, and inductive interactions. We formalize and instantiate four primitives, translating abstract transition dynamics into concrete interactive environments. Building upon this, we establish OdysseyArena-Lite for standardized benchmarking, providing a set of 120 tasks to measure an agent's inductive efficiency and long-horizon discovery. Pushing further, we introduce OdysseyArena-Challenge to stress-test agent stability across extreme interaction horizons (e.g., > 200 steps). Extensive experiments on 15+ leading LLMs reveal that even frontier models exhibit a deficiency in inductive scenarios, identifying a critical bottleneck in the pursuit of autonomous discovery in complex environments. Project homepage: https://os-copilot.github.io/OdysseyArena-Home/
Figures & tables
Figure 1: Comparison between deductive and inductive settings in multi-turn agentic tasks.
Benchmark
Ind.
Horizon
Deploy
BabyAI [ 2019 ]
✓
Long
Simulator
ALFWorld [ 2021 ]
✓
Short
Simulator
GAIA [ 2023 ]
✗
Long
Offline
WebArena [ 2024 ]
✗
Short
Docker
OSWorld [ 2024 ]
✗
Long
Docker
AndroidWorld [ 2025 ]
✗
Short
Emulator
Table 1: Comparison of representative multi-turn agentic benchmarks. Ind. indicates whether inductive reasoning is required. Horizon denotes the number of steps required to complete a task, categorized as short ( < 50), long (50–100), or X-Long ( > 100). Deploy describes the evaluation setup required to run the environment, with API-based deployment being the most lightweight.
Figure 2: Demonstrations of four OdysseyArena environments: Turn On Lights, AI Trading, Energy Dispatch, and Repo System. For clarity, we omit the task prompts here and present only the interaction trajectories. Full prompts are provided in Appendix C .
Figure 3: Overview of the benchmark architecture, illustrating the environment configuration initialization (left) and the interaction loop between the LLM agent and the environment step logic (right).
Model
Turn On Lights
AI Trading
Energy Dispatch
Repo System
Avg@4
Pass@4
Avg@4
Best@4
Avg@4
Pass@4
Avg@4
Pass@4
Human
81.67
100.00
+92.55%
+197.23%
25.00
60.00
77.50
100.00
Gemini 3 Pro Preview
44.17
76.67
+67.71%
+76.94%
30.00
36.67
65.83
80.00
GPT-5
28.33
40.00
+17.32%
+20.47%
23.33
40.00
62.50
83.33
Gemini 2.5 Pro
29.17
50.00
+33.02%
+40.12%
10.83
26.67
50.00
66.67
gpt-oss-120b (high)
27.50
40.00
+23.27%
+27.47%
0.00
0.00
18.33
33.33
Table 2: Performance comparison on four environments. We provide three different reasoning effort of gpt-oss-120b. For AI Trading environment, we report the profit rate and Best@4 is calculated based on the highest profit of each task. For other three environments, we report the success rate. Colored Rows represent proprietary models. The best results are in bold .
Figure 6Figure 7
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Environment
Action
Description
Turn On Lights
Toggle <light>
The parameter light is the specific index of the light you want to toggle. This action is successfully executed only if its latent logical condition is satisfied.
Purchase specific shares of stock and sell specific shares of stock. Multiple stocks can be acquired or sold simultaneously. The buy action is successful only if the cash is enough. If the selling shares exceed the current holdings, the environment defaults to selling the entire existing position.
Dispatch p 1 , p 2 , p 3 units of thermal, wind, and solar power respectively. For p 4 , a negative value represents charging and positive represents discharging. Notably, charging is truncated upon reaching the battery’s max capacity, and the discharging is limited to the available capacity if it exceeds the current state of charge. The cost of the four energies should be limited within the total budget.
Repo System
repo tree/ls
Inspect the repository’s directory structure to identify script paths.
pip install <package>
Install the package . Supports specific version ( == ) and range constraints ( > , >= , < , <= ).
pip uninstall <package>
Remove the specified package from the configuration.
Appendix
Table 4: Action Space of and description for OdysseyArena . All parameters are wrapped in <> .
Model
Turn On Lights
AI Trading
Energy Dispatch
Repo Management
w/o rules
w/ rules
w/o rules
w/ rules
w/o rules
w/ rules
w/o rules
w/ rules
Gemini 3 Pro Preview
44.17
100.00
+67.71%
+135.48%
30.00
16.67
65.83
96.67
GPT-5
28.33
100.00
+17.32%
+132.02%
23.33
13.33
62.50
96.67
Gemini 2.5 Pro
29.17
100.00
+33.02%
+126.24%
10.83
20.00
50.00
96.67
gpt-oss-120b (high)
27.50
100.00
+23.27%
+141.63%
0.00
0.00
18.33
50.00
DeepSeek-V3.2
18.33
96.67
+8.62%
+118.88%
0.00
0.00
48.33
56.67
Appendix
Table 5: Performance comparison between w/o rules and w/ rules settings. We provide three different reasoning effort of gpt-oss-120b. For AI Trading environment, we report the profit rate and pass@4 is calculated based on the highest profit of each task. For other three environments, we report the success rate. Colored Rows represent proprietary models.
Figure 7: All data for Success Rate against Step. We do not plot for Energy Dispatch environment due to its complex success conditions.
Figure 9: Task success status (based on pass@4 ). Each row represents: (a) Human, (b) Gemini3 Pro Preview, (c) GPT-5, (d) Gemini 2.5 Pro, (e) gpt-oss-120b (high), (f) DeepSeek-V3.2, (g) Grok 4 Fast, (h) Qwen3-235B-A22B-Instruct, (i) gpt-oss-120b (medium), (j) Qwen3-30B-A3B-Instruct, (k) GLM-4-32B-0414, (l) gpt-oss-120b (low), (m) Llama 3.3 70B Instruct, (n) Qwen3-4B-Instruct, (o) Llama 3.1 8B Instruct, (p) GLM-4-9B-Chat. Dark green cells indicate tasks solved by Human. Green cells indicate tasks solved by LLM agents. Gray cells indicate unsolved tasks. We report the average success rate in Repo System across all LLMs for each subset (Easy, Medium and Hard).
Model
Avg@4
Pass@4
200 turns
150 turns
100 turns
50 turns
200 turns
150 turns
100 turns
50 turns
Gemini 3 Pro Preview
65.83
70.00
76.67
70.83
80.00
83.33
86.67
83.33
gpt-oss-120b-high
18.33
19.17
23.33
11.67
33.33
33.33
36.67
20.00
Appendix
Table 6: Performance in Repo System with different memory usage.
Model
Original Avg@4
In-Context Avg@4
Original Pass@4
In-Context Pass@4
Gemini 3 Pro Preview
65.83
66.67
80.00
81.67
gpt-oss-120b-high
18.33
17.50
33.33
30.00
Appendix
Table 7: In-Context Learning Performance Comparison in Repo System
Figure 10: The line chart represents token usage and the bar chart represents token efficiency. Token Usage is measured in units of 106 , while Token Efficiency is reported in units of 10−6 .
Table 10: Inter-rater reliability analysis across four experimental environments. For tasks with discrete success metrics ( Turn On Lights , Energy Dispatch , and Repo System ), we report Fleiss’ Kappa ( κ ). For the continuous reasoning task ( AI Trading ), the Intraclass Correlation Coefficient (ICC) is employed.
LLM agents are increasingly deployed in long-horizon, complex environments to solve challenging problems, but this expansion exposes them to long-horizon attacks that exploit multi-turn user-agent-environment interactions to achieve objectives infeasible in single-turn settings. To measure agent vulnerabilities to such risks, we present AgentLAB, the first benchmark dedicated to evaluating LLM agent susceptibility to adaptive, long-horizon attacks. Currently, AgentLAB supports five novel attack types including intent hijacking, tool chaining, task injection, objective drifting, and memory poisoning, spanning 28 realistic agentic environments, and 644 security test cases. Leveraging AgentLAB, we evaluate representative LLM agents and find that they remain highly susceptible to long-horizon attacks; moreover, defenses designed for single-turn interactions fail to reliably mitigate long-horizon threats. We anticipate that AgentLAB will serve as a valuable benchmark for tracking progress on securing LLM agents in practical settings. The benchmark is publicly available at https://tanqiujiang.github.io/AgentLAB_main.
Tanqiu Jiang, Yuhui Wang, Jiacheng Liang +1
Department of Computer Science, Stony Brook University, Stony Brook, USA.
Large Language Models (LLMs) struggle with long-horizon tasks due to the "context bottleneck" and the "lost-in-the-middle" phenomenon, where accumulated noise from verbose environments degrades reasoning over multi-turn interactions. To address this issue, we introduce a symbiotic framework that decouples context management from task execution. Our architecture pairs a lightweight, specialized policy model, ContextCurator, with a powerful frozen foundation model, TaskExecutor. Trained via reinforcement learning, ContextCurator actively reduces information entropy in the working memory. It aggressively prunes environmental noise while preserving reasoning anchors, that is, sparse data points that are critical for future deductions. On WebArena, our framework improves the success rate of Gemini-3.0-flash from 36.4% to 41.2% while reducing token consumption by 8.8% (from 47.4K to 43.3K). On DeepSearch, it achieves a 57.1% success rate, compared with 53.9%, while reducing token consumption by a factor of 8. Remarkably, a 7B ContextCurator matches the context management performance of GPT-4o, providing a scalable and computationally efficient paradigm for autonomous long-horizon agents.
Xiaozhe Li, Tianyi Lyu, Yizhao Yang +6
Tongji University · CurrentsAI Research · Stanford University +1
Large Language Model (LLM) agents are increasingly deployed in unfamiliar and evolving environments, requiring them to proactively explore, understand, and adapt to environmental dynamics. However, existing benchmarks rarely test whether agents can proactively discover how an environment works and adapt as it evolves. To address this gap, we introduce AdaExploreBench, a benchmark for proactive exploration and test-time adaptation in evolving environments. By instantiating environmental dynamics as hidden rules, AdaExploreBench requires agents to actively acquire evidence and infer the rules. Under implicit rule drift, the latent rule changes without announcement, requiring agents to detect from degraded performance that prior knowledge is stale and re-explore. Experiments on eight leading LLMs show that proactive and adaptive exploration remain challenging: GPT-6 Sol achieves only 61.9% calibrated accuracy, while providing sufficient evidence or removing rule drift improves average performance significantly by close to 10%. These results reveal a critical bottleneck in current LLM agents: they can often reason effectively from informative evidence once it is available, but struggle to actively acquire the right evidence and recognize when previously acquired evidence is no longer valid.