Language-model agents differ fundamentally from prior intelligent systems in their ability to learn from natural language in context. This capability enables agents to iteratively refine their answers in response to feedback, a prevalent mode of operation in modern agentic workflows. However, the community primarily ranks models by metrics such as Pass@1 or Pass@K, which cannot capture how they utilize feedback on prior attempts. We introduce Seq@K, a natural evaluation protocol that measures whether a model can solve a problem within K sequential attempts, provided feedback after each previous failure. Unlike multi-turn agentic benchmarks that examine long-horizon planning, Seq@K isolates and formalizes the core capability of refinement from feedback. We evaluate Seq@K on a suite of 7 benchmarks and 16 frontier models, and show that Seq@K reveals a materially different model-selection landscape: under Seq@K, (1) model rankings often differ, (2) the Pareto frontiers admit more efficient configurations and (3) the gains from choosing the optimal model configuration can widen or shrink relative to those of Pass@K. We further show that analyzing model trajectories under Seq@K helps characterize iterative refinement behavior, such as the rate at which models address issues raised in feedback, or how rigidly they hold onto their own answers. Finally, we study how the nature of feedback affects models' iterative refinement, showing that practitioners must carefully design and benchmark the feedback loops of iterative agentic systems. Together, these results establish how Seq@K offers a more faithful metric for intelligent systems that typically operate in such feedback loop, and the necessity to consider Seq@K in evaluating models for these systems.
Figures & tables
Benchmark
Actors
Runs
Used in
AdvancedIF ( He et al., 2025 )
10
61
Sections 5.1 , 5.2 , 5.3
CLBench DKR ( Dou et al., 2026 )
8
80
Sections 5.1 , 5.2 , 5.3
CLBench RSA ( Dou et al., 2026 )
3
8
Section 5.1
HealthBench ( Arora et al., 2025 )
7
67
Sections 5.1 , 5.3
ResearchRubrics ( Sharma et al., 2025 )
8
35
Sections 5.1 , 5.3
MBPP Pro ( Yu et al., 2025 )
4
112
Sections 5.1 , 5.2 , 5.3
Table 1: Experimental coverage across evaluation cohorts. Overview of completed run records associated with our analyses across the 7 evaluated benchmark variants. This table summarizes the actor coverage, run counts, and the Results sections each benchmark is associated with. The Actors column counts distinct models evaluated across benchmarks.
Figure 1: Feedback changes model ordering. (a) AdvancedIF: 7 actors, 30 shared tasks, raw feedback. (b) DKR: 6 actors, 28 shared tasks, actor-specific critic configurations. All use T=0.7 . Rank trajectories run from Pass@5 through Seq@1–5, colored by success rate. The number below each point is the rank change from the previous column. Shaded groups are tied and share the group’s best rank; order within a tie is arbitrary, and a move within a tie is no rank change. These are descriptive orderings. Full scores and configurations: Appendix C .
Figure 2: Short sequential calibration picks better actors for sequential deployment. Selection loss and calibration cost for n=1 – 20 calibration tasks ( T=0.7 ). The picked actor is evaluated under Seq@5 with raw verifier feedback (containing details on failed tests). Markers show n∈{1,5,10,20} , and Table 2 reports n=10 . On MBPP Pro, Seq@3 calibration on 1 task reaches about the same selection loss as Pass@5 on 10 tasks at 11× lower cost. Vertical arrows mark the largest gap between Seq@3 and Pass@5 among the plotted task counts. Error bars are 95% bootstrap intervals. The dashed line is a uniformly random actor choice baseline, which incurs no calibration cost.
Mean cost (USD)
Chosen actor’s Seq@5 (%)
Benchmark
K
Pass@5
Seq@ K
Pass@5 pick
Seq@ K pick
Gain (pp)
MBPP Pro
3
0.407
0.383
90.2
98.4
+8.3
MBPP Pro
4
0.407
0.432
90.2
98.2
+8.1
HumanEval Pro
3
1.015
0.805
70.8
96.8
+26.0
HumanEval Pro
4
1.015
0.937
70.8
96.7
+25.9
Table 2: Selection with 10 calibration tasks. The mean calibration cost per selection and the held-out Seq@5 of the picked actor, with T=0.7 on 45 MBPP Pro and 40 HumanEval Pro tasks. Seq@K calibration uses K=3 or 4 . Note that both Seq@K and Pass@K stop at the first success without incurring additional cost. Gain shows the improvement from better picked actor from calibrating with Seq@5 compared to that of Pass@5 .
Best–worst model gap
All-pairs gap
Benchmark
Pass@5 → Seq@5
Change
Change
AdvancedIF
33.3→6.7
−26.7
−11.7
MBPP Pro
18.5→31.1
+12.6
+8.6
HumanEval Pro
10.8→51.7
+40.8
+25.4
Table 3: Pass@5 can overstate or understate the performance gap between actors. The gap between the best and worst actor and the average gap over all model pairs, with T=0.7 . Seq@5 uses raw verifier feedback on AdvancedIF and details on failed tests on MBPP Pro and HumanEval Pro. Change is the gap under Seq@5 minus the gap under Pass@5 . Details are provided in Appendix B.2 .
Figure 3: Sequential feedback expands cost–accuracy frontiers. Each frontier shows the highest success reached for a given mean cost per task, for K=1 to 5 with T=0.7 . Both Seq@K and Pass@K stop incurring cost at the first success. Labeled markers are on the frontier and show the actor and K , and faint markers are not on the frontier. Seq@K tends to reveal a better Pareto frontier in both (a) AdvancedIF across 7 actors, 30 shared tasks, and raw verifier feedback and (b) DKR across 6 actors, 28 shared tasks.
Figure 4: Final success correlates across protocols, but gains from retries do not. Each point is one model–benchmark pair with K=5 . Seq@5 uses the best non-binary feedback configuration for each pair. (a) Seq@5 against Pass@5 . (b) The gain from attempts 2 to 5 under each protocol. Details are in Appendix C .
Actor
Feedback / critic
Seq@1
Seq@5
EGS
LGS
Opus 4.7
Directive / actor itself
0.000
0.500
0.33
0.07
DeepSeek V4 Pro
Socratic / GPT-5.2
0.067
0.400
0.30
0.10
Gemini 3 Flash
Socratic / GPT-5.2
0.033
0.267
0.86
0.00
GPT-5.2
Directive / actor itself
0.033
0.300
0.25
0.00
Qwen3.6 Max
Socratic / GPT-5.2
0.033
0.367
0.20
0.00
Table 4: Some configurations revise earlier than others. On selected CLBench DKR configurations and 30 tasks. EGS and LGS breakdown when each actor successfully revise their attempts, with higher EGS potentially more suitable where earlier revisions are preferred, and non-zero LGS indicating the need for more sequential attempts allocated. More details are provided in Appendix D.1 .
Figure 5: Revisions do not always address the feedback. (a) The fraction of revised attempts in coding that are exactly the same as the previous attempt, for four models on HumanEval Pro and MBPP Pro. Each point is one of three independent runs on each benchmark, and the line shows the average over all six runs. DeepSeek and Qwen models appear more likely to submit revisions identical to the previous failed attempts. (b) The success rate of the next attempt, grouped by whether the revision is unhelpful, addresses some, or addresses all of the feedback. The issue of breaking previously passing criteria is particularly pronounced in the DKR and AdvancedIF benchmarks, where success requires meeting multiple criteria, and less so the case in coding benchmarks.
Condition
Actor receives
Failed criteria repaired (%)
Independent sampling
No earlier answer or feedback
36.5
Binary feedback
Earlier answers + failure verdict
34.4
Rubric-blind critic
Critique without grading criteria
36.8
Rubric-aware critic
Critique informed by grading criteria
73.8
Table 5: In DKR, feedback is more useful when the critic knows what is being graded. The fraction of requirements failed in the first attempt of DKR that are fixed in a later attempt, pooled over five models with 30 tasks each. Binary feedback and critiques without the rubric repair about as many requirements as independent retries. Details are in Appendix F.1 .
Benchmark
Trajectory pairs
Change (pp)
AdvancedIF
200
−0.5
MBPP Pro
800
+1.8
HumanEval Pro
800
+0.5
CLBench DKR
180
−6.1
ResearchRubrics
180
−7.2
HealthBench
360
−14.4
Table 6: Summarizing the history can lower sequential success. Here, each unit is a pair of trajectories on the same task where one has previous attempts summarized, and one keeps the full history. The two trajectories otherwise use the same model, feedback, grader, and attempt budget. The table shows the average change in success rate from summarization over these pairs. Performance changes little on AdvancedIF and coding but drops on the other benchmarks.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Evaluated task set
Success criteria
AdvancedIF ( He et al., 2025 )
Prepared instruction-following tasks
All required instruction verdicts pass
CLBench DKR / RSA ( Dou et al., 2026 )
Domain Knowledge Reasoning / Rule System Application
Full-task rubric success
HealthBench ( Arora et al., 2025 )
Hard split, context-seeking and uncertainty themes
Normalized score at least 0.75 , plus negative-criterion and safety gates
Table 7: Benchmarks and success criteria. The task set and success criterion of each benchmark.
Benchmark
K
Cost ratio [95% CI]
Gain (pp) [95% CI]
MBPP Pro
3
0.94[0.83,1.08]
+8.3[+1.8,+19.9]
MBPP Pro
4
1.06[0.90,1.26]
+8.1[+1.5,+19.7]
HumanEval Pro
3
0.79[0.68,0.91]
+26.0[+6.8,+44.7]
HumanEval Pro
4
0.92[0.76,1.12]
+25.9[+7.0,+44.1]
Appendix
Table 8: Seq@3 and Seq@4 against Pass@5 with 10 calibration tasks. Cost ratio is the mean calibration cost of Seq@K calibration divided by that of Pass@5 calibration. Gain is how much higher the picked actor’s Seq@5 is with Seq@K calibration than with Pass@5 calibration, in percentage points. Brackets are 95% bootstrap intervals.
Benchmark
Best–worst gap change
All-pairs gap change
AdvancedIF
−26.7[−43.3,−3.3]
−11.7[−21.0,−1.6]
MBPP Pro
+12.6[−7.4,+29.6]
+8.6[−2.8,+18.5]
HumanEval Pro
+40.8[+12.5,+52.5]
+25.4[+10.0,+31.9]
Appendix
Table 9: Change in the gap between actors from Pass@5 to Seq@5 , with 95% intervals. Values are in points, as in Table 3 .
Actor
Pass@5
Seq@5
Opus 4.7
0.071
0.500
DeepSeek V4 Pro
0.143
0.429
Gemini 3 Flash
0.036
0.250
GPT-5.2
0.143
0.286
Qwen3.6 27B
0.000
0.357
Qwen3.6 Max
0.071
0.393
Appendix
Table 10: DKR scores on the 28 tasks of Figure 3 . Six actors, T=0.7 , seed 42. The sequential critic differs across actors (Appendix C.1 ). Table 11 uses 30 tasks and averages DeepSeek V4 Pro over four seeds, so its scores differ.
Benchmark
Actor
N
Pass@1
Pass@5
Seq@1
Seq@5
AdvancedIF
Opus 4.7
30
0.433
0.633
0.467
1.000
Sonnet 4.6
30
0.367
0.500
0.400
1.000
DeepSeek V4 Pro
30
0.433
0.667
0.567
0.967
Gemini 3 Flash
30
0.483
0.783
0.533
0.967
GPT-5.2
30
0.400
0.550
0.400
0.933
Qwen3.6 27B
30
0.633
0.800
0.667
1.000
Appendix
Table 11: Scores of the 26 model–benchmark pairs in Figure 4 . N is the number of shared tasks. All use T=0.7 . Seq@K uses the non-binary feedback setting with the highest Seq@5 for each pair.
Actor
Seq@1
Seq@2
Seq@3
Seq@4
Seq@5
Opus 4.7
0.000
0.167
0.267
0.467
0.500
DeepSeek V4 Pro
0.067
0.167
0.333
0.367
0.400
Gemini 3 Flash
0.033
0.233
0.233
0.267
0.267
GPT-5.2
0.033
0.100
0.133
0.300
0.300
Qwen3.6 Max
0.033
0.100
0.300
0.367
0.367
Appendix
Table 12: Seq@K on DKR by attempt. Seed 42, 30 tasks per actor. The feedback setup differs across actors (Table 4 ).
Setting
Feedback; critic; grader
Actors (runs)
Tasks
Revisions
AdvancedIF, o3-mini
raw verifier output; none; o3-mini
Gemini 3 Flash, GPT-5.2 ( K=5 ); Opus 4.7, GPT-5.5 Pro ( K=3 )
55
153
AdvancedIF, Sonnet
raw verifier output; none; Sonnet 4.6
8 actors, 13 runs (Sonnet ×5 , Haiku ×2 )
21
256
CLBench DKR
rubric-aware socratic; GPT-5.2; GPT-5.2
DeepSeek V4 Pro, Gemini 3 Flash, Qwen3.6 27B, Qwen3.6 Max
Table 13: Settings of the revision annotation. Tasks and revisions count held-out tasks only.
Case / task index
Actor
Feedback
Judge
Attempts
Hestia / 4
Qwen Max
blind critique
GPT-5.4
2
Hestia / 4
Qwen Max
binary
GPT-5.4
5
DJ / 12
Gemini
blind critique
o3-mini
5
DJ / 12
Gemini
raw
o3-mini
2
SideHustleBot / 13
Qwen Max
raw
GPT-5.4
3
SideHustleBot / 13
Gemini
raw
o3-mini
4
Appendix
Table 14: Trajectories we used. The judge is the one set for the run. Task indices are positions within the run’s benchmark, not dataset IDs.
Actor
Independent
Binary
Rubric-blind critic
Rubric-aware critic
Opus 4.7
37.2 / 1.0
36.9 / 0.7
26.0 / 1.9
77.1 / 2.3
DeepSeek V4 Pro
39.1 / 2.4
33.9 / 1.4
44.0 / 3.0
74.6 / 2.1
Gemini 3 Flash
30.8 / 4.2
28.5 / 3.1
42.0 / 3.5
69.8 / 3.0
GPT-5.2
39.4 / 1.7
39.7 / 1.7
37.4 / 3.4
76.1 / 1.7
Qwen3.6 Max
37.9 / 2.3
34.8 / 2.4
30.7 / 2.6
72.2 / 1.3
Pooled
36.5 / 2.2
34.4 / 1.8
36.8 / 2.9
73.8 / 2.1
Appendix
Table 15: Criterion repair and regression on DKR by actor (%). Each cell is repair / regression. Seed 42, 30 tasks per actor. The rubric-aware critics are those of Table 4 , and the rubric-blind critic gives socratic feedback.
Benchmark
Actor
Pass@5
Binary Seq@5
Test-detail Seq@5
MBPP Pro
Sonnet 4.6
0.637
0.807
0.985
MBPP Pro
DeepSeek V3.2
0.593
0.585
0.770
MBPP Pro
Gemini 3 Flash
0.526
0.622
0.978
MBPP Pro
Qwen3 Coder Flash
0.452
0.481
0.674
HumanEval Pro
Sonnet 4.6
0.358
0.617
0.975
HumanEval Pro
DeepSeek V3.2
0.342
0.375
0.575
Appendix
Table 16: Coding scores with K=5 . Each cell averages three runs at T=0.7 on 45 MBPP Pro or 40 HumanEval Pro tasks.
Benchmark
Trajectory pairs
Actors
Tasks
K
Change (pp)
HealthBench
360
2
60
5
−14.4
CLBench DKR
180
2
30
5
−6.1
ResearchRubrics
180
4
30
5, 10
−7.2
AdvancedIF
200
4
50
10
−0.5
MBPP Pro
800
4
50
5, 10
+1.8
HumanEval Pro
800
4
50
5, 10
+0.5
Appendix
Table 17: Change in success rate when the history is summarized, in percentage points, averaged over trajectory pairs. Each trajectory is scored at its run’s attempt budget K . Coding runs use T=0 , and the others use T=0.7 .
AI agent research spans a wide spectrum: from RL agents that learn from scratch to foundation model agents that leverage pre-trained knowledge, yet no unified benchmark enables fair comparison across these approaches. We present Agentick, a benchmark for sequential decision-making agents designed to evaluate RL, LLM, VLM, hybrid, and human agents on common ground and to power research on the fundamental challenges of sequential decision-making. Agentick provides 37 procedurally generated tasks across six capability categories, four difficulty levels, and five observation modalities, all exposed through a single Gymnasium-compatible interface. The benchmark ships with a Coding API, oracle reference policies for all tasks, pre-built SFT datasets, a composable agent harness, and a live leaderboard. An evaluation spanning 27 configurations and over 90,000 episodes reveals that no single approach dominates: GPT-5 mini leads overall at 0.309 oracle-normalized score while PPO dominates planning and multi-agent tasks; the reasoning harness multiplies LLM performance by 3-10x; and ASCII observations consistently outperform natural language. These findings highlight the substantial room for improvement that remains across all agent paradigms. Agentick's capability-decomposed, multi-modal design provides the empirical infrastructure needed to drive progress toward general autonomous agents, both as an evaluation framework and as a training ground for RL post-training of foundation models in truly sequential environments.
Roger Creus Castanyer, Pablo Samuel Castro, Glen Berseth
Mila Quebec AI Institute · Université de Montréal · Mila Quebec AI Institute Université de Montréal +1
It is increasingly important to evaluate the characteristics of systems based on large language models (LLMs). Evaluations in this context often rely on a curated benchmark set of input prompts provided to the LLM, where the output for each prompt may be assigned a binary or ordinal score and the aggregation of scores across prompts is then used as a summary evaluation. In this paper, we develop a Bayesian approach for quantifying the uncertainty that arises in such evaluation metrics as a result of the stochasticity of the LLM-based systems; the same prompt may exhibit different outcomes on repeated runs. Our framework leads naturally to a sequential evaluation, in which we leverage the Bayesian model to preferentially select which prompts in the benchmark to use next, enabling more cost-effective LLM evaluations. We demonstrate this approach through four case studies: pairwise LLM preferences in interactive dialogue (MT-Bench), refusal rates on adversarial inputs (JailbreakBench), embodied agentic reasoning (ALFWorld), and agentic web-search Q&A (GAIA). Our results illustrate how a Bayesian approach can provide insightful uncertainty quantification about the behavior of blackbox LLM-based systems in a manner that is not accessible via deterministic or single-run evaluations.
Saatvik Kher, Shang Wu, Rachel Longjohn +2
Department of Computer Science, University of California, Irvine · Department of Statistics, University of California, Irvine
Evaluating language models and agents requires combining benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality. Differences in interfaces, metric semantics, and trace formats complicate these comparisons and can hide failures in agentic workflows. We present AuditKIT, a source-available evaluation library that records these measurements in one result representation, with direction-aware metric contracts, configuration fingerprints, and extensible metrics and backends. For agents, it provides turn-aware tool-call matching, separates parallel from sequential execution, and distinguishes verified task outcomes from trajectory-level diagnostics. Missing or unobservable evidence is marked unscored rather than counted as failure. For retrieval-augmented generation, it combines ranked retrieval metrics, judged grounding, and deterministic evidence checks. Seven case studies spanning model comparison, serving runtimes, retrieval, tool calling, and interoperability show how answer extraction, runtime behavior, and metric selection change evaluation outcomes.