ResearchGym: Evaluating Language Model Agents on Real-World AI Research

Aniketh Garikaparthi; Manasi Patwardhan; Arman Cohan

cs.AIFeb 16, 2026

ResearchGym: Evaluating Language Model Agents on Real-World AI Research

Authors: Aniketh Garikaparthi, Manasi Patwardhan, Arman Cohan

Organizations: TCS Research · Yale University

Abstract

We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research. To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL. From each paper's repository, we preserve the datasets, evaluation harness, and baseline implementations but withhold the paper's proposed method. This results in five containerized task environments comprising 39 sub-tasks in total. Within each environment, agents must propose novel hypotheses, run experiments, and attempt to surpass strong human baselines on the paper's metrics. In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap. The agent improves over the provided baselines from the repository in just 1 of 15 evaluations (6.7%) by 11.5%, and completes only 26.5% of sub-tasks on average. We identify recurring long-horizon failure modes, including impatience, poor time and resource management, overconfidence in weak hypotheses, difficulty coordinating parallel experiments, and hard limits from context length. Yet in a single run, the agent surpasses the solution of an ICML 2025 Spotlight task, indicating that frontier agents can occasionally reach state-of-the-art performance, but do so unreliably. We additionally evaluate proprietary agent scaffolds including Claude Code (Opus-4.5) and Codex (GPT-5.2) which display a similar gap. ResearchGym provides infrastructure for systematic evaluation and analysis of autonomous agents on closed-loop research.

Figures & tables

Figure 1: ResearchGym combines the aspects of ideation and experimentation , evaluating LLM agents in executable research codebases with objective scores. rg-agent (w/ GPT-5): (A) Best@3 normalized performance, averaged over all primary sub-tasks, shaded region represents a 95% Confidence Interval generated via percentile bootstrapping. (B) depicts the number of sub-tasks completed. (C) shows mean normalized performance over all primary sub-tasks. Error bars represent the min–max range (3 runs). Metrics defined in (§ 2.4 ).

Benchmark

source

eval.

uncon.

live.

div.

crea.

dev.

loop.

fea.

gpu.

time.

budget.

Research Ideation

Future-Idea-Generation ( Kumar et al., 2024 )

Papers

LLM Judge

✗

✓

✗

–

IdeaBench ( Guo et al., 2024 )

Papers

LLM Judge

✓

✗

✓

✗

–

ResearchBench ( Liu et al., 2025b )

Papers

LLM Judge

✓

✗

✓

✗

–

AI Idea Bench 2025 ( Qiu et al., 2025 )

Papers

LLM Judge

✓

✗

–

Machine Learning Engineering

Table 1: Comparison across relevant benchmarks on key aspects ( ✓ = present, ✗ = absent).

Figure 2: Benchmark Construction Pipeline: LLMs are used to generate compact task cards from award-winning papers. After two-stage filtering, each paper’s repository is manually cleaned and finalized into a benchmark task. Benchmark: Consists of 5 curated tasks and 39 sub-tasks across diverse domains, a sub-task is typically validating the proposed method under different datasets/settings.

Paper	Abbrv	Conference	Category	Evaluation Metric
Incorporating Domain Knowledge into Materials Tokenization ( Oh et al., 2025 )	mdt	ACL SAC Highlights	NLP, Tokenization, Physical Sciences	Micro-F1, Macro-F1
Test-time Adaptation for Cross-modal Retrieval with Query Shift ( Li et al., 2025 )	cmr	ICLR Spotlight	Multimodality, Information Retrieval	Text-to-Image Recall, Image-to-Text Recall
TIMING: Temporality-Aware Integrated Gradients for Time Series Explanation ( Jang et al., 2025 )	tim	ICML Spotlight	Time Series, Explainability	CPD, AUP, AUR
SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning ( Wu et al., 2025b )	cl	ICLR Oral	Meta Learning, Continual Learning	Accuracy, Average Anytime Accuracy
Prioritized Generative Replay ( Wang et al., 2025 )	irb	ICLR Oral	Reinforcement Learning	Average Returns

Table 2: Selected papers for ResearchGym . All conferences are from year 2025 .

	CL		MDT		CMR		TIM		IRB
	Acc ( $\uparrow$ )	AAA ( $\uparrow$ )	Macro-F1 ( $\uparrow$ )	Micro-F1 ( $\uparrow$ )	I2TR@1 ( $\uparrow$ )	T2IR@1 ( $\uparrow$ )	CPD (A) ( $\uparrow$ )	CPD (Z) ( $\uparrow$ )	Return ( $\uparrow$ )
rg-agent	$30.75_{\text{{{±37.39}}}}$	$43.17_{\text{{{±35.81}}}}$	$13.95_{\text{{{±24.16}}}}$	$14.08_{\text{{{±24.40}}}}$	$37.42_{\text{{{±24.00}}}}$	$46.57_{\text{{{±40.00}}}}$	$0.329_{\text{{{±0.276}}}}$	$0.337_{\text{{{±0.252}}}}$	$579.79_{\text{{{\color[rgb]{0.5,0.5,0.5}±585.47}}}}$
best@3	$80.42_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.07}}}}$	$86.02_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.37}}}}$	41.85	42.25	58.95	70.00	$0.589_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.000}}}}$	$0.525_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.000}}}}$	$1407.06_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.000}}}}$
baseline	$86.75_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.35}}}}$	$91.72_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.15}}}}$	$84.40_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.3}}}}$	$83.50_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.4}}}}$	60.65	69.9	$0.448_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.013}}}}$	$0.573_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.022}}}}$	$3395.21_{\text{{{\color[rgb]{0.5,0.5,0.5}±117.50}}}}$
sota	$88.01_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.31}}}}$	$92.54_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.18}}}}$	$85.35_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.3}}}}$	$84.90_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.4}}}}$	63.35	70.30	$0.463_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.007}}}}$	$0.602_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.033}}}}$	$4101.79_{\text{{{\color[rgb]{0.5,0.5,0.5}±244.05}}}}$

Table 3: Task native performance on all primary sub-tasks . rg-agent (w/ GPT-5 as base model) row shows mean and std. deviation across 3 runs. Other rows show per-task mean and std. deviation statistics. Green cells indicate the LLM performance could exceed the baseline , ext are best performing runs with additional resources. baseline and sota are results from the source paper.

Task	Normalized Performance		Completed	Attempts	Init Time	Cost	Tokens (M)
	avg	best@3			(min)	(usd)	in	out	reason
cl	$0.4033_{\text{{{±0.39}}}}$	$0.9429_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.0}}}}$	✓✓✓	$11.33_{\text{{{\color[rgb]{0.5,0.5,0.5}±4.93}}}}$	$85.67_{\text{{{\color[rgb]{0.5,0.5,0.5}±25.58}}}}$	$2.00_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.77}}}}$	$5.41_{\text{{{\color[rgb]{0.5,0.5,0.5}±2.87}}}}$	$0.08_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.02}}}}$	$0.06_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.02}}}}$
mdt	$0.1650_{\text{{{±0.23}}}}$	$0.4939_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.0}}}}$	✓ ✗✗	$12.67_{\text{{{\color[rgb]{0.5,0.5,0.5}±11.35}}}}$	$33.41_{\text{{{\color[rgb]{0.5,0.5,0.5}±0}}}}$	$2.78_{\text{{{\color[rgb]{0.5,0.5,0.5}±2.14}}}}$	$14.51_{\text{{{\color[rgb]{0.5,0.5,0.5}±3.16}}}}$	$0.20_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.03}}}}$	$0.16_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.03}}}}$
cmr	$0.6260_{\text{{{±0.46}}}}$	$0.9630_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.0}}}}$	✗ ✓✓	$5.66_{\text{{{\color[rgb]{0.5,0.5,0.5}±2.52}}}}$	$110.00_{\text{{{\color[rgb]{0.5,0.5,0.5}±96.71}}}}$	$9.66_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.59}}}}$	$31.84_{\text{{{\color[rgb]{0.5,0.5,0.5}±8.54}}}}$	$0.30_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.11}}}}$	$0.25_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.09}}}}$
tim	$0.6351_{\text{{{±0.50}}}}$	$1.0721_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.0}}}}$	✓✓✓	$8.00_{\text{{{\color[rgb]{0.5,0.5,0.5}±4.36}}}}$	$252.33_{\text{{{\color[rgb]{0.5,0.5,0.5}±71.29}}}}$	$5.50_{\text{{{\color[rgb]{0.5,0.5,0.5}±2.09}}}}$	$19.93_{\text{{{\color[rgb]{0.5,0.5,0.5}±10.05}}}}$	$0.25_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.09}}}}$	$0.20_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.07}}}}$
irb	$0.1414_{\text{{{±0.14}}}}$	$0.3430_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.0}}}}$	✓✓✓	$4.33_{\text{{{\color[rgb]{0.5,0.5,0.5}±2.64}}}}$	$14.33_{\text{{{\color[rgb]{0.5,0.5,0.5}±4.72}}}}$	$1.62_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.93}}}}$	$1.63_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.69}}}}$	$0.06_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.02}}}}$	$0.04_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.01}}}}$

Table 4: Results and statistics on the primary sub-task and corresponding resource consumptions. Results for runs using rg-agent (w/ GPT-5), reporting mean and std. deviation across 3 independent runs. All columns pertain to the primary sub-task.

Task	Avg. Norm.	Completion	Improvement	Tool Call
	Performance	Rate	Rate	Success
cl	$0.4033_{\text{{{±0.39}}}}$	$16.67_{\text{{{±00.00}}}}$	✗	$86.86_{\text{{{±5.71}}}}$
mdt	$0.1650_{\text{{{±0.23}}}}$	$27.76_{\text{{{±48.15}}}}$	✗	$84.34_{\text{{{±3.64}}}}$
cmr	$0.6260_{\text{{{±0.46}}}}$	$26.18_{\text{{{±12.14}}}}$	✗	$86.12_{\text{{{±3.59}}}}$
tim	$0.6351_{\text{{{±0.28}}}}$	$28.57_{\text{{{±30.86}}}}$	✓	$84.18_{\text{{{±3.99}}}}$
irb	$0.1414_{\text{{{±0.14}}}}$	$33.33_{\text{{{±24.00}}}}$	✗	$83.10_{\text{{{±0.44}}}}$

Table 5: Aggregate reliability (per task). Completion : mean sub-task completion rate (valid grades / total sub-tasks). Improve : fraction of runs that beat the strongest provided baseline on the primary sub-task (optionally require

\geq\epsilon

margin). Tool Success : % of actions which did not result in errors (incl. execution errors).

Figure 3: Performance vs. Resources: Plots depict the relationship between best performance and consumed resources across all tasks. The overall trend shows a weak but positive correlation among the two, with diminishing returns.

Figure 4: Performance vs. Tool Usage: rg-agent : (A) Illustrates efficient allocation of reasoning tokens to respective tools, (B) Demonstrates a natural exploration/exploitation paradigm overtime with respective tool usage mix, (C) Establishes a moderate negative correlation between action density and performance, and (D) Further shows the diminishing increase in performance with resources.

Task	Claude Code		Codex		RG-Hint		RG-Async
	Normalized	Overall	Normalized	Overall	Normalized	Overall	Normalized	Overall
	Performance	Completion	Performance	Completion	Performance	Completion	Performance	Completion
cl	0.985	100%	0.979	17%	0.885	17%	0.660	17%
mdt	–	33%	0.490	83%	0.829	83%	–	0%
cmr	–	50%	0.966	80%	0.928	43%	–	14%
tim	–	0%	0.171	100%	–	71%	–	14%

Table 6: Performance across scaffold variations. Normalized Performance : normalized primary sub-task’s score. Overall Completion : sub-task completion rate (valid grades / total sub-tasks). (–) indicates the agent could not produce a valid submission.

Failure mode	Description
Overconfidence in weak hypotheses	Agents often commit to a method without basic sanity checks such as baseline replication. This leads to confident iteration on foundations that were never validated.
Optimization and myopia	Runs frequently drift into surface-level tuning (e.g., hyperparameters) even when signals suggest larger objective-level changes are needed. This produces many trials with little to no improvement.
Non-comparable experiments	A common pattern is changing multiple factors at once or altering evaluation setup. This leads to the agent continuously iterating without a signal of whether it is actually improving.
Impatience and premature convergence	After finding the first runnable approach, agents tend to keep patching that line of attack instead of branching to alternatives. This reduces exploration, increasingly getting stuck in local optima.
Poor time and resource management	Agents launch expensive runs before validating correctness, or fail to reserve wall-time for grading and controlled comparisons. They also fail to optimally utilize GPUs.
Parallel experiment collapse	Agents are poor at starting and maintaining parallel experiments. They also lack a reliable track of what was tried, what failed, and what remains open; compounding into further confusion.

Table 7: Failure modes and long-horizon limitations observed in agent interaction traces.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Category	System	Description
	AlphaEvolve ( Novikov et al., 2025 )	Evolutionary coding agent for algorithm discovery/optimization across math and computing (data-center scheduling, chip design, kernels, math). Reports: +0.7% Borg recovery, 23% kernel speedup ( $\Rightarrow$ 1% LLM train time cut), up to 32.5% FlashAttention speedup, $\sim$ 20% of $\sim$ 50 open math problems improved.
Evolutionary Search	OpenEvolve ( Sharma, 2025 )	Open-source AlphaEvolve-style evolutionary code search; shown on GPU/kernel optimization, circle packing, algorithm design; repo reports $\sim$ 2.8 $\times$ kernel speedups (Apple M1 Pro) and SOTA circle packing at $n{=}26$ .
	ADRS (AI-Driven Research for Systems) ( Cheng et al., 2025b ; Cheng et al., 2025a )	Automating systems research (e.g., scheduling, load balancing) via iterative LLM-generated code and simulator-based scoring; case studies report ADRS-generated algorithms matching/exceeding human SOTA.
	Automated Idea Executor ( Si et al., 2026 )
	AI-Scientist ( Yamada et al., 2025 )	Tree-based planning and execution, through parallel experiment generation and iterative debugging. One full execution resulted in a paper which passed workshop level peer-review at a top-tier conference.
Tree-based Search	AIDE ( Jiang et al., 2025 )	Tree-based exploration of the solution space. AIDE iteratively draft programs as nodes in a tree, debugging and improving. Current demonstrated SOTA on RE-Bench.

Appendix

Table 8: Systems related to automated / agentic research, grouped by control strategy.

Paper	Conference	Category	Evaluation metric
NeuronTune: Towards Self-Guided Spurious Bias Mitigation ( Zheng et al., 2025 )	ICML 2025	Deep Learning, Robustness	Worst Group Accuracy, Accuracy Gap
Multi-Label Test-Time Adaptation with Bound Entropy Minimization ( Wu et al., 2025a )	ICLR 2025	VLMs, Test-Time Adaptation	mean Average Precision (mAP)
Learning Monotonic Probabilities with a Generative Cost Model ( Tang et al., 2025c )	ICML 2025	Generative Models and Autoencoders	MAE, AUC, Acc, RMSE

Appendix

Table 9: Selected papers for ResearchGym (Development Set) .

Title	Abbrv	arXiv (v1)	Citations	GitHub stars	License	GPU requirements
Incorporating Domain Knowledge into Materials Tokenization ( Oh et al., 2025 )	MDT	2025-06-09	0	2	CC By 4.0	None
Test-time Adaptation for Cross-modal Retrieval with Query Shift ( Li et al., 2025 )	CMR	2024-10-21	17	28	Apache 2.0	None
TIMING: Temporality-Aware Integrated Gradients for Time Series Explanation ( Jang et al., 2025 )	TIM	2025-06-05	0	12	CC By 4.0	None
SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning ( Wu et al., 2025b )	CL	2025-01-22	24	64	MIT License	None
Prioritized Generative Replay ( Wang et al., 2025 )	IRB	2024-10-23	8	21	MIT License	12GB
NeuronTune: Towards Self-Guided Spurious Bias Mitigation ( Zheng et al., 2025 )	SBM	2025-05-29	1	2	CC By 4.0	None

Appendix

Table 10: Task metadata for ResearchGym . Citations and GitHub stars as of 2025-10-10 . arXiv (v1) dates denote initial preprint submission. GPU requirements are reported if mentioned in the paper and reproduced manually for verification.

Candidate paper	Reason for exclusion	Principles
ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features ( Helbling et al., 2025 )	Passed keyword-based filtering, but closer inspection revealed substantial GPU requirements, making it infeasible within our target compute budget.	P1
SURGEON: Memory-Adaptive Fully Test-Time Adaptation via Dynamic Activation Sparsity ( Ma et al., 2025 )	Despite strong fit (objective evaluation, low VRAM, no large datasets), the reported metrics (GFLOPs, cache, latency) were hardware-native and tied to a specific edge system (Jetson Xavier NX). Running on A100 would invalidate baselines and confound comparisons.	P2
Learning Class Prototypes for Unified Sparse-Supervised 3D Object Detection ( Zhu et al., 2025b )	Datasets were gated (email/forms) with licensing/access friction, and exceeded 300GB, making execution impractical within our 12–24 hour budget.	P1, P3
Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection ( Yan et al., 2025b )	Dependent on gated datasets of prohibitive size, creating both accessibility and execution-time barriers.	P1, P3
Overcoming Shortcut Problem in VLM for Robust Out-of-Distribution Detection ( Xu et al., 2025 )	Training and evaluation could not reliably complete within our 12–24 hour execution budget due to dataset scale and end-to-end runtime.	P1
FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation ( Zhang et al., 2025a )	Loading required 7B-scale models exceeded our VRAM budget; introducing quantization would alter baseline performance and compromise comparability, so we excluded it to preserve consistency.	P1

Appendix

Table 11: Representative candidates excluded after manual audit. (P1: compute/runtime/VRAM budget; P2: hardware-tied metrics/baselines; P3: dataset accessibility/licensing/size constraints.)

Task	Ambiguities encountered during task packaging
cl	utils/inc_net.py contains methods which can potentially bias agent’s towards LoRa ideas, we retained due to baseline relevance.
cmr	Removed ’tcr’ branches and witheld certain overlapping .yaml configs, edited default flags, trimmed novel loss functions.
tim	Removed 10 .sh scripts while preserving essential utils and templates, external baseline dirs mentioned ’our’ but were preserved.
mdt	A vocab_mappings.txt file was retained as it contained important mappings for relevant chemical symbols.
irb	Removed manuscripts, edited entrypoints, wrappers around relevant methods while retaining an important submodule.

Appendix

Table 12: Brief example of task-packaging ambiguities for included ResearchGym tasks. More details in repository.

Tool	Purpose	Examples
bash	Run shell commands inside the target environment; used for installing deps, listing files, running training / eval scripts.	ls -l data/ ; pip install -r requirements.txt ; sh run_exp.sh
python	Execute inline Python for quick checks, small data processing, or calling library functions without leaving the agent loop.	inspect JSON output; validate metrics; run inference.py --config cfg.yaml
read-file-chunk	Read (parts of) large files without loading everything; useful for long logs, notebooks, or codebases.	read first 200 lines of train.log ; inspect a single module; peek at error trace
search-file	Keyword / pattern search within files to locate relevant functions, classes, parameters etc.	find where Trainer is defined; locate config.yaml ; search for “accuracy” in logs
apply-patch	Edit files programmatically using patches; ensures deterministic edits and makes multi-step refactors easy.	add new argument to main.py ; fix import path; update model hyperparams
write_file	Write new content to a specific file path, creating parent directories and overwriting existing files after a user-approved diff.	create configs/new_exp.yaml ; overwrite src/model.py with updated implementation; dump generated report to results/summary.txt

Appendix

Table 13: Tools exposed to the ResearchGym agent during final execution.

Source	cl	tim	mdt	irb	cmr	total
arXiv	13	10	10	13	11	57
Official Proceedings	5	5	4	4	3	21
GitHub	4	3	2	2	5	16
Mirrors/Archives	6	6	6	7	6	31
Academic Aggregators	2	5	4	3	2	16
Author/Project Pages	4	2	3	4	6	19

Appendix

Table 14: Blocked URLs by category across tasks

	Inspection verdicts					Inspection cost / usage (GPT-5)
Scaffold	Runs	Pass	Suspicious	Review	Inspected	Input (M)	Output (K)	Cost ($)	Time (min)	$/run
BasicAgent	25	10	11	2	25	32.6	547	18.19	221	0.73
Claude Code	7	1	0	6	7	4.2	145	2.76	35	0.39
Codex	10	2	0	8	10	9.0	171	4.50	52	0.45
Synthetic tests	6	2	4	0	6	3.0	98	2.75	23	0.46
Total	48	15	15	16	48	48.8	961	28.20	331	0.59

Appendix

Table 15: Inspection verdicts and GPT-5 inspection-agent resource usage by agent scaffold. Inspected

=

Pass

+

Suspicious

+

Review. Manual review found all Suspicious flags corresponded to genuine cheating (no false positives), but some cheating may have gone unflagged (false negatives).

	ImageNet - R (N=5)		ImageNet - R (N=10)		ImageNet - R (N=20)		ImageNet - A (N=10)
Method	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$
Full Fine - Tuning	$64.92_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.87}}}}$	$75.57_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.50}}}}$	$60.57_{\text{{{\color[rgb]{0.5,0.5,0.5}±1.06}}}}$	$72.31_{\text{{{\color[rgb]{0.5,0.5,0.5}±1.09}}}}$	$49.95_{\text{{{\color[rgb]{0.5,0.5,0.5}±1.31}}}}$	$65.32_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.84}}}}$	$16.31_{\text{{{\color[rgb]{0.5,0.5,0.5}±7.89}}}}$	$30.04_{\text{{{\color[rgb]{0.5,0.5,0.5}±13.18}}}}$
L2P	$73.04_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.71}}}}$	$76.94_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.41}}}}$	$71.26_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.44}}}}$	$76.13_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.46}}}}$	$68.97_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.51}}}}$	$74.16_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.32}}}}$	$42.94_{\text{{{\color[rgb]{0.5,0.5,0.5}±1.27}}}}$	$51.40_{\text{{{\color[rgb]{0.5,0.5,0.5}±1.95}}}}$
DualPrompt	$69.99_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.57}}}}$	$72.24_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.41}}}}$	$68.22_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.20}}}}$	$73.81_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.39}}}}$	$65.23_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.45}}}}$	$71.30_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.16}}}}$	$45.49_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.96}}}}$	$54.68_{\text{{{\color[rgb]{0.5,0.5,0.5}±1.24}}}}$
CODA - Prompt	$76.63_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.27}}}}$	$80.30_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.28}}}}$	$74.05_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.41}}}}$	$78.14_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.39}}}}$	$69.38_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.33}}}}$	$73.95_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.63}}}}$	$45.36_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.78}}}}$	$57.03_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.94}}}}$
HiDe - Prompt	$74.77_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.25}}}}$	$78.15_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.24}}}}$	$74.65_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.14}}}}$	$78.46_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.18}}}}$	$73.59_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.19}}}}$	$77.93_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.19}}}}$	$42.70_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.60}}}}$	$56.32_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.40}}}}$

Appendix

Table 16: Baseline results on ImageNet-R (N=5,10,20) and ImageNet-A (N=10); values are mean with std as subscript.

	CIFAR100		CUB200
Method	Acc $\uparrow$	AAA $\uparrow$	Acc $\uparrow$	AAA $\uparrow$
Full Fine - Tuning	$69.49_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.50}}}}$	$80.35_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.87}}}}$	$51.43_{\text{{{\color[rgb]{0.5,0.5,0.5}±1.41}}}}$	$69.74_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.93}}}}$
L2P	$83.18_{\text{{{\color[rgb]{0.5,0.5,0.5}±1.20}}}}$	$87.69_{\text{{{\color[rgb]{0.5,0.5,0.5}±1.05}}}}$	$65.18_{\text{{{\color[rgb]{0.5,0.5,0.5}±2.49}}}}$	$76.12_{\text{{{\color[rgb]{0.5,0.5,0.5}±1.27}}}}$
DualPrompt	$81.48_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.86}}}}$	$86.41_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.66}}}}$	$68.00_{\text{{{\color[rgb]{0.5,0.5,0.5}±1.06}}}}$	$79.40_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.88}}}}$
CODA - Prompt	$86.31_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.12}}}}$	$90.67_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.22}}}}$	$71.92_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.33}}}}$	$78.76_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.65}}}}$
InfLoRA	$86.75_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.35}}}}$	$91.72_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.15}}}}$	$70.82_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.23}}}}$	$81.39_{\text{{{\color[rgb]{0.5,0.5,0.5}±0.14}}}}$

Appendix

Table 17: Results on CIFAR100 (primary) and CUB200. Paper’s Method (SD-LoRA) was withheld from the agent.

		Generation Task
Tokenization	Metric	NER	RC	EAE	PC	SAR	SC	SF	Overall
	Micro-F1	${55.7}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.4}}}$	${49.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${48.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.8}}}$	${67.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$	${61.1}_{\text{{\color[rgb]{0.5,0.5,0.5}±1.8}}}$	${90.7}_{\text{{\color[rgb]{0.5,0.5,0.5}±2.4}}}$	${36.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±1.4}}}$	${63.5}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.5}}}$
BPE ( Sennrich et al., 2016 )	Macro-F1	${47.1}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.5}}}$	${47.2}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.9}}}$	${36.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.3}}}$	${40.2}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.0}}}$	${41.8}_{\text{{\color[rgb]{0.5,0.5,0.5}±1.3}}}$	${47.6}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.0}}}$	${16.7}_{\text{{\color[rgb]{0.5,0.5,0.5}±1.6}}}$	${42.0}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.9}}}$
	Micro-F1	${76.6}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${80.9}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.3}}}$	${48.5}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${73.1}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.5}}}$	${81.9}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.4}}}$	${90.0}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$	${57.4}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${72.6}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$
WordPiece ( Wu et al., 2016 )	Macro-F1	${56.1}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${58.5}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.6}}}$	${29.4}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.3}}}$	${58.9}_{\text{{\color[rgb]{0.5,0.5,0.5}±1.0}}}$	${74.6}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.9}}}$	${60.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.8}}}$	${32.6}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${52.9}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$
	Micro-F1	${77.0}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${82.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.4}}}$	${47.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$	${68.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.8}}}$	${77.1}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.4}}}$	${90.9}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$	${57.1}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.3}}}$	${71.4}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$

Appendix

Table 27

		Classification Task
		NER ${}_{\text{SOFC}}$		NER ${}_{\text{Matscholar}}$		SF		RC		PC*
Tokenization	Metric	val	test	val	test	val	test	val	test	val	test
	Micro-F1	${81.6}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${81.4}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$	${86.4}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.3}}}$	${84.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.5}}}$	${68.1}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.5}}}$	${68.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.6}}}$	${90.2}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.4}}}$	${89.9}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.0}}}$
BPE ( Sennrich et al., 2016 )	Macro-F1	${80.7}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${78.9}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$	${85.0}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.6}}}$	${82.9}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.7}}}$	${65.5}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.4}}}$	$\text{59.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.8}}}$	${86.4}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$	${85.5}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$	${95.5}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.0}}}$	${95.6}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.0}}}$
	Micro-F1	${82.0}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.6}}}$	${80.9}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.4}}}$	${88.8}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${86.1}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.3}}}$	${67.4}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.5}}}$	$\textbf{60.4}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.7}}}$	${90.6}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${91.0}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.7}}}$
WordPiece ( Wu et al., 2016 )	Macro-F1	${83.0}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${83.0}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.4}}}$	${87.6}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.3}}}$	${85.8}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.2}}}$	${69.2}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.4}}}$	${69.6}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.4}}}$	${86.3}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.3}}}$	${87.5}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$	${95.2}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$	${95.2}_{\text{{\color[rgb]{0.5,0.5,0.5}±0.1}}}$

Appendix

Table 28

	Noise				Blur				Weather				Digital
Query Shift	Gauss.	Shot	Impul.	Speckle	Defoc.	Glass	Motion	Zoom	Snow	Frost	Fog	Brit.	Contr.	Elastic	Pixel	JPEG	Avg.
BLIP ViT-B/16	43.4	46.3	43.2	57.3	43.3	68.0	39.7	8.4	32.3	52.2	57.0	66.8	36.0	41.3	20.6	63.7	45.0
$\bullet~$ Tent	41.6	40.5	37.9	54.0	44.7	65.1	39.6	8.3	31.9	48.7	56.3	66.5	31.8	40.3	19.2	62.3	43.0
$\bullet~$ EATA	41.4	50.3	35.7	63.1	49.8	72.2	46.2	6.9	45.6	56.7	62.5	71.4	43.6	51.3	25.6	67.0	49.3
$\bullet~$ SAR	42.3	51.5	37.5	61.8	40.3	71.5	32.8	6.2	38.0	56.2	59.1	70.6	31.1	53.5	17.5	66.4	46.0
$\bullet~$ READ	45.8	48.4	37.2	59.9	44.5	71.8	46.6	11.5	39.9	49.9	58.4	70.3	35.8	45.0	18.8	66.2	46.9

Appendix

Table 18: Comparisons with state-of-the-art methods on COCO-C benchmark under query shift on the image modality with maximum severity level regarding the Recall@1 metric. The best results are marked in bold .

	Character-level					Word-level					Sentence-level
Query Shift	OCR	CI	CR	CS	CD	SR	RI	RS	RD	IP	Formal	Casual	Passive	Active	Backtrans	Avg.
BLIP ViT-B/16	31.4	11.3	9.4	18.9	11.4	43.6	51.5	50.3	50.6	56.8	56.6	56.2	54.9	56.8	54.2	40.9
$\bullet~$ Tent	31.4	11.0	9.5	17.7	11.3	43.2	51.3	50.3	50.6	56.6	56.2	56.0	54.9	56.9	53.9	40.7
$\bullet~$ EATA	33.1	11.9	10.5	18.4	12.0	44.9	53.0	51.6	50.3	56.2	56.8	56.8	56.0	56.8	54.3	41.5
$\bullet~$ SAR	31.8	11.6	9.9	18.5	11.7	43.6	51.5	50.3	50.6	56.8	56.5	56.2	54.9	56.8	54.2	41.0
$\bullet~$ READ	32.3	11.4	9.6	18.2	11.2	44.3	52.9	51.7	51.1	57.6	57.1	56.7	55.9	57.1	54.7	41.4

Appendix

Table 19: Comparisons with state-of-the-art methods on COCO-C benchmark under query shift on the text modality with maximum severity level regarding the Recall@1 metric.

	Base2Flickr		Base2COCO		Base2Fashion		Base2Nocaps(ID)		Base2Nocaps(ND)		Base2Nocaps(OD)
Query Shift	TR@1	IR@1	TR@1	IR@1	TR@1	IR@1	TR@1	IR@1	TR@1	IR@1	TR@1	IR@1	Avg.
CLIP ViT-B/16	80.2	61.5	52.5	33.0	8.5	13.2	84.9	61.4	75.4	49.2	73.8	55.8	54.1
$\bullet~$ Tent	81.4	64.0	48.8	27.6	5.6	10.7	85.1	61.7	74.6	48.6	71.8	56.1	53.0
$\bullet~$ EATA	80.4	63.4	52.1	34.8	8.1	12.0	84.7	62.0	75.1	52.3	74.1	56.9	54.7
$\bullet~$ SAR	80.3	62.2	51.8	33.9	8.0	13.3	84.7	61.3	75.4	51.3	73.7	56.1	54.3
$\bullet~$ READ	80.6	64.4	46.0	35.7	5.8	11.2	85.1	63.0	75.0	52.1	73.5	57.0	54.1

Appendix

Table 20: Comparisons with state-of-the-art methods on benchmarks under Query-Gallery shifts regarding the Recall@1 metric. In the table, “ID", “ND" and “OD" refer to “In-Domain", “Near-Domain" and “Out-Domain", respectively. Besides, “TR@1" / “IR@1" represent Recall@1 for image-to-text retrieval / text-to-image retrieval.

	CUHK2ICFG	ICFG2CUHK
Query Shift	IR@1	IR@1	Avg.
CLIP ViT-B/16	33.3	41.0	37.2
$\bullet~$ Tent	33.5	41.9	37.7
$\bullet~$ EATA	33.3	42.2	37.8
$\bullet~$ SAR	33.3	42.2	37.8
$\bullet~$ READ	33.0	42.3	37.7

Appendix

Table 21: Comparisons with state-of-the-art methods on ReID benchmarks under Query-Gallery shifts regarding the Recall@1 metric.

MIMIC-III

PAM

Boiler

Epilepsy

Wafer

Freezer

Method

Avg.

Zero

Avg.

Zero

Avg.

Zero

Avg.

Zero

Avg.

Zero

Avg.

Zero

AFO

0.127

\pm

0.009

0.227

\pm

0.017

0.140

\pm

0.009

0.200

\pm

0.013

0.262

\pm

0.020

0.349

\pm

0.035

0.028

\pm

0.003

0.030

\pm

0.004

0.018

\pm

0.003

0.018

\pm

0.003

0.143

\pm

0.054

0.143

\pm

0.054

GradSHAP

0.250

\pm

0.015

0.522

\pm

0.038

0.421

\pm

0.014

0.518

\pm

0.012

0.752

\pm

0.055

0.747

\pm

0.092

0.052

\pm

0.004

0.054

\pm

0.004

0.485

\pm

0.014

0.485

\pm

0.014

0.397

\pm

0.110

0.397

\pm

0.110

Extrmask

0.154

\pm

0.008

0.305

\pm

0.010

0.291

\pm

0.007

0.380

\pm

0.009

0.338

\pm

0.028

0.400

\pm

0.031

0.028

\pm

0.003

0.029

\pm

0.003

0.202

\pm

0.026

0.202

\pm

0.026

0.176

\pm

0.057

0.176

\pm

0.057

ContraLSP

0.048

\pm

0.003

0.051

\pm

0.004

0.046

\pm

0.007

0.059

\pm

0.011

0.408

\pm

0.035

0.496

\pm

0.043

0.016

\pm

0.001

0.016

\pm

0.001

0.121

\pm

0.032

0.121

\pm

0.032

0.176

\pm

0.055

0.176

\pm

0.055

TimeX++

0.017

\pm

0.002

0.074

\pm

0.006

0.057

\pm

0.004

0.070

\pm

0.004

0.124

\pm

0.028

0.208

\pm

0.043

0.030

\pm

0.004

0.032

\pm

0.004

0.000

\pm

0.000

\pm

0.000

0.216

\pm

0.056

0.216

\pm

0.056

Appendix

Table 22: Performance comparison of various XAI methods on real-world datasets with 10% feature masking. Results are aggregated as mean

\pm

standard error over five random cross-validation repetitions and presented across multiple datasets, including MIMIC-III, PAM, Boiler (Multivariate), Epilepsy, Wafer, and Freezer (Univariate). Evaluation metrics include cumulative prediction difference (CPD) attribution performance under two feature substitution strategies: average substitution (Avg.) and zero substitution (Zero).

	Switch-Feature
Method	CPD $\uparrow$	AUP $\uparrow$	AUR $\uparrow$
FO	0.191 $\pm$ 0.006	0.902 $\pm$ 0.009	0.374 $\pm$ 0.006
AFO	0.182 $\pm$ 0.007	0.836 $\pm$ 0.012	0.416 $\pm$ 0.008
GradSHAP	0.196 $\pm$ 0.006	0.892 $\pm$ 0.010	0.387 $\pm$ 0.006
DeepLIFT	0.196 $\pm$ 0.007	0.918 $\pm$ 0.019	0.432 $\pm$ 0.011
LIME	0.195 $\pm$ 0.006	0.949 $\pm$ 0.015	0.391 $\pm$ 0.016

Appendix

Table 23: Performance comparison of various XAI methods on Switch Feature and State datasets. Results are reported as mean

\pm

standard error over five cross-validation repetitions, evaluated using AUP, AUR, and CPD (10% masking) for true saliency map and cumulative masking strategies.

Time	PAM Avg CPD	PAM Zeros CPD
4h 57m	0.231	0.333
5h 16m	0.203	0.369
5h 35m	0.175	0.367
5h 55m	0.103	0.590
6h 21m	0.070	0.582
6h 44m	0.115	0.115

Appendix

Table 24: Run 002 score progression on the PAM dataset (primary sub-task). The agent iteratively improved through 13 evaluation attempts, demonstrating sustained experimental discipline.

	DMC-100k (Online)				Pixel-DMC-100k (Online)
Environment	Quadruped- Walk	Cheetah- Run	Reacher- Hard	Finger-Turn- Hard*	Walker- Walk	Cheetah- Run
mbpo	505.91 $\pm$ 252.55	450.47 $\pm$ 132.09	777.24 $\pm$ 98.59	631.19 $\pm$ 98.77	-	-
dreamer-v3	389.63 $\pm$ 168.47	362.01 $\pm$ 30.69	807.58 $\pm$ 156.38	745.27 $\pm$ 90.30	353.40 $\pm$ 114.12	298.13 $\pm$ 86.37
sac	178.31 $\pm$ 36.85	346.61 $\pm$ 61.94	654.23 $\pm$ 211.84	591.11 $\pm$ 41.44	-	-
redq	496.75 $\pm$ 151.00	606.86 $\pm$ 99.77	733.54 $\pm$ 79.66	520.53 $\pm$ 114.88	-	-
redq + curiosity	687.14 $\pm$ 93.12	682.64 $\pm$ 52.89	725.70 $\pm$ 87.78	777.66 $\pm$ 116.96	-	-

Appendix

Table 25: Average returns on state and pixel-based DMC after 100K environment steps (5 seeds, 1 std. dev. err.). * is a harder environment with sparser rewards, and so we present results over 300K timesteps.

	Walker2d-v2	HalfCheetah-v2	Hopper-v2
mbpo	3781.34 $\pm$ 912.44	8612.49 $\pm$ 407.53	3007.83 $\pm$ 511.57
dreamer-v3	4104.67 $\pm$ 349.74	7126.84 $\pm$ 539.22	3083.41 $\pm$ 138.90
sac	879.98 $\pm$ 217.52	5065.61 $\pm$ 467.73	2033.39 $\pm$ 793.96
redq	3819.17 $\pm$ 906.34	6330.85 $\pm$ 433.47	3275.66 $\pm$ 171.90
synther	4829.32 $\pm$ 191.16	8165.35 $\pm$ 1534.24	3395.21 $\pm$ 117.50
(Curiosity)	5682.33 $\pm$ 370.04	9234.61 $\pm$ 658.77	4101.79 $\pm$ 244.05

Appendix

Table 26: Results on state-based OpenAI gym tasks. We report average return after 100K environment steps. Results are over 3 seeds, with 1 std. dev. err.

Explore similar work

Jul 6, 2026

AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments

Language agents, i.e., LLM agents, progress rapidly and are increasingly deployed in production environments. This trend underscores the urgent need for rigorous and realistic evaluations. However, most existing benchmarks evaluate agents in simplified, idealized settings. They typically rely on pre-packaged tool interfaces, overlook critical steps, and assume inputs are clean and fully specified. Consequently, they understate the difficulty of real deployments, where uncertainty and noise are ubiquitous and agents must proactively explore the environment to uncover new tools. To bridge this gap, we present AgentGym2, a new evaluation framework with task instances grounded in real-world end-to-end working demands. Beyond reasoning and planning, it measures agents' ability to execute end-to-end procedures, discover tools via exploration, compose tools for unseen tasks, and remain robust to noisy and underspecified information. Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2, revealing a substantial gap between the capability of current agents and the demands of real-world applications.

Zhiheng Xi, Dingwen Yang, Jiaqi Liu +22

Fudan University · Zhejiang University · Shanghaijiaotong University +1

LLM Agent Evaluation AI Agent Benchmarks LLM Agent Reliability

Ask PDF

May 16, 2026

1GC-7RC: One Graphic Card -- Seven Research Challenges! How Good Are AI Agents at Doing Your Job?

Autonomous AI coding agents are becoming a core tool for ML practitioners in industry and research alike. Despite this growing adoption, no standardized benchmark exists to evaluate their ability to design, implement, and train models from scratch across diverse domains. We introduce 1GC-7RC (Single Graphic Card: Seven Research Challenges), a benchmark comprising seven ML tasks spanning language modeling, image classification, semantic segmentation, graph learning, tabular prediction, time-series forecasting, and text classification. Each task provides a locked data-preparation and evaluation script together with a baseline training script; the agent may only modify the training code, has no access to pretrained weights (with one controlled exception for semantic segmentation), no internet access, and must complete each task within a task-specific wall-clock budget (40-120 minutes) on a single GPU. We evaluate seven coding agents: five proprietary (Claude Code with Sonnet 4.6, Opus 4.6, and Opus 4.7; Codex CLI with GPT 5.5; and OpenCode with Qwen 3.6+) and two open-source (OpenCode with Kimi K2.5, Kimi K2.6). Across 5 runs per agent-task pair, we report substantial performance differences that reveal varying levels of implicit ML knowledge, planning ability, and time-budget management. The benchmark, harness, and all evaluation artifacts are publicly available on GitHub at https://github.com/Strolchii/1GC-7RC-Benchmark to facilitate reproducible comparison of future agents. Because our benchmark design is modular, the benchmark can be extended to new tasks and domains, adapted to different GPU budgets, and used to study multi-agent settings, making it a flexible platform for future research on autonomous research agents.

Robin-Nico Kampa, Fabian Deuser, Anna Bößendörfer +2

University of the Bundeswehr Munich, Munich, Germany

AI Coding Agents AI Agent Evaluation AI Agent Benchmarks Autonomous Research Agents

Ask PDF

Jun 5, 2026

Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle

As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Despite their evolution from research assistants into autonomous research agents, these systems still exhibit significant limitations in field sensitivity, research ethics, and nuanced scientific judgment. Consequently, frontier agents remain unable to fully replace human researchers. To bridge this gap, we conceptualize the AARR (Act As a Real Researcher) benchmark series. Unlike existing benchmarks that primarily assess macro-level execution capabilities, AARR focuses on whether agents can emulate the professionalism, thoroughness, and nuanced reasoning that characterize human researchers in granular research scenarios. In this work, we propose AARRI-Bench (Act As a Real Research Intern), the first benchmark in this series. We conduct extensive experiments across frontier models and agentic systems, revealing that even the best-performing configuration (Mini-SWE-Agent with Claude Opus 4.7) achieves only 68.3% success rate, frequently overlooking subtle yet critical details that are obvious to real human researchers. Our results indicate that developing researcher-like AI requires further exploration of research behavior, rather than merely complex scaffolding. Our data is released at https://github.com/AARR-bench/AARRI-bench.

Jiayu Wang, Weijiang Lv, Bowen Fu +8

Xi’an Jiaotong University · Xidian University

AI-Assisted Scientific Research AI Agent Evaluation AI Agent Benchmarks Autonomous Research Agents

Ask PDF