Interactive Benchmarks

Baoqing Yue; Zihan Zhu; Yutong Han; Brian Fan; Qian Sun; Jichen Feng; Hufei Yang; Yifan Zhang

cs.AIMar 5, 2026

Interactive Benchmarks

Authors: Baoqing Yue, Zihan Zhu, Yutong Han, Brian Fan, Qian Sun, Jichen Feng, Hufei Yang, Yifan Zhang,

Organizations: InteractiveBench · Princeton University

Abstract

Existing reasoning evaluation paradigms suffer from different limitations: fixed benchmarks are increasingly saturated and vulnerable to contamination, while preference-based evaluations rely on subjective judgments. We argue that a core aspect of intelligence is the ability to decide what information to acquire and how to use it effectively. We propose Interactive Benchmarks, a unified evaluation paradigm that assesses a model's reasoning ability through budgeted multi-turn interaction. We evaluate models under this framework in two settings: Interactive Proofs, where models interact with a judge to solve Logic, UI2Html, and Mathematics tasks under objective feedback; and Interactive Games, where models reason strategically to maximize long-horizon utilities. Our results show that interactive benchmarks provide a more robust assessment of this dimension of model intelligence, revealing substantial room for improvement in interactive scenarios.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the Interactive Benchmarks Framework. Interactive benchmarks act as a sequential interaction process. Left: In Interactive Proofs, the Player queries a Judge to converge on an objective truth by minimizing uncertainty. Right: In Interactive Games, the Player acts in a stochastic or adversarial environment to maximize long-term utility.

Metric	Grok	Gemini	GPT-5	Kimi	DeepSeek	Qwen3
Direct-answer accuracy	0.00%	0.00%	0.00%	0.00%	0.00%	0.00%

Table 1: Accuracy in the no-interaction setting. Each model directly submits a final answer from the initial narrative without querying the Judge. All evaluated models achieve

0\%

accuracy, indicating that the task requires interaction to recover the hidden explanation.

Panel of Figure 2 — Figure 2: Evaluation results on Situation Puzzle . We report both success rate and interaction efficiency under a fixed $20$ -turn budget with a fixed judge.

Figure 3: Comparison between the full interaction setting and a single-round baseline. Each pair of bars corresponds to the same player model.

Panel of Figure 4 — Figure 4: Interactive vs. pass@ $k$ evaluation under the same budget constraint. Left: accuracy under interactive evaluation and pass@ $k$ , plus the underestimation gap. Right: interaction efficiency measured by average rounds among correct trials.

Metric	Grok	Gemini	GPT-5	Kimi	DeepSeek	Qwen3
Interactive Token Usage	279.06	3749.96	479.10	3430.19	6608.13	15802.67
Pass@1 Token Usage	208.88	686.52	244.37	1702.38	1741.48	1991.92
Budget-matched $k$	1	5	2	2	4	8

Table 2: Budget-matched pass@k statistics. For each model,

k^{\star}

is chosen by the closest-budget rule in Eq. ( 3 ); pass@

k

is therefore approximately budget matched to the interactive protocol.

Figure 5: Comparison of six LLM poker agents across 10 independent tables (bars: mean, error bars: standard deviation). (a) Average winnings per hand, (b) VPIP rate, (c) response latency, and (d) fold rate.

Figure 6: Trust Game tournament results measured by average payoff per round. Dashed horizontal lines denote the Grim Trigger and TFT heuristic baselines.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Logic judge ablation on the Situation Puzzle benchmark. The x-axis varies the judge model. Left: success rate. Right: average turns over solved instances only.

Figure 8: Logic interaction-budget ablation on the Situation Puzzle benchmark. The x-axis denotes the maximum number of interaction rounds, and the y-axis reports success rate.

Figure 9: UI2Html judge ablation. The x-axis varies the shared judge stack used for summarization, comparison, and final scoring, while each bar denotes a different player model. The y-axis reports average overall reconstruction score.

Panel of Figure 10 — Figure 10: Math judge ablation on the HLE subset. Rows correspond to evaluated players and columns correspond to judges. Left: accuracy. Right: average number of questions among solved instances only.

Figure 11: Trust Game ablation over the continuation probability

\delta

. The x-axis shows the implied expected number of rounds

1/(1-\delta)

. Left: average score per round. Right: cooperation rate.

Figure 12: Trust Game behavioral statistics. Left axis (blue bars) shows cooperation rate; right axis (pink bars) shows betrayal rate.

Figure 13: Example UI2Html target webpage for GPT-5-mini . The figure shows the hidden reference solution corresponding to the initial textual prompt.

Panel of Figure 14 — Figure 14: Example UI2Html reconstruction case for Geminire . Top: the hidden reference solution corresponding to the initial textual prompt. Bottom left: the first-round rendered webpage generated from the initial prompt. Bottom right: the final rendered webpage produced after iterative interaction and refinement.

Reasoning Evaluation

Reasoning evaluation measures whether AI systems solve multi-step problems reliably, use evidence and valid intermediate steps, and generalize beyond benchmarks.

See all 51 papers

Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games
May 26, 2026Mingyuan Fan, Weiguang Han, Daixin Wang +3LLM Evaluation AI Agent Benchmarks
East China Normal University · Ant Group
GIM: Evaluating models via tasks that integrate multiple cognitive domains
May 18, 2026Rohit Patel, Alexandre Rezende, Steven McClainLLM Evaluation Test-Time Scaling
Meta Superintelligence Labs
Interactive Evaluation Requires a Design Science
May 18, 2026Keyang Xuan, Peiyang Song, Pan Lu +10LLM Agent Evaluation AI Agent Evaluation
University of Texas Austin · California Institute of Technology · Carnegie Mellon University +5

May 26, 2026J/K move · Enter open · S save

Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games

Mingyuan Fan, Weiguang Han, Daixin Wang, Cen Chen+2

East China Normal University · Ant Group

LLM Evaluation AI Agent Benchmarks Language Model Robustness LLM Reasoning

Open paper AskPDF arXiv

We introduce a multi-turn interactive framework for reasoning evaluation that treats reasoning as active evidence acquisition and belief updating. Wherein, LLMs receive only the task rules, must issue targeted queries to a hidden environment, integrate partial observations over time, and decide when to submit a final answer. Beyond standard success rate and interaction efficiency, we evaluate contextual robustness under controlled contextual perturbations, and metacognitive adaptation through counterfactual revision and necessity judgment. We instantiate the framework as a benchmark of 474 executable games, each evaluated under five fixed configuration search spaces corresponding to five difficulty levels, and evaluate a broad set of frontier LLMs. Results show that the benchmark is highly discriminative, exposing large differences not only in success rate but also in interaction efficiency. Moreover, we empirically show that contextual perturbations cause moderate but consistent declines, whereas counterfactual revision and necessity judgment lead to much larger drops.

Abstract

Reasoning Evaluation

Explore similar work

Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games

GIM: Evaluating models via tasks that integrate multiple cognitive domains

Interactive Evaluation Requires a Design Science