ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation

Xingshan Zeng; Lingzhi Wang; Weiwen Liu; Liangyou Li; Yasheng Wang; Lifeng Shang; Xin Jiang; Qun Liu

cs.CLFeb 2, 2026

ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation

Authors: Xingshan Zeng, Lingzhi Wang, Weiwen Liu, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu

Organizations: Huawei Technologies Co., Ltd · Harbin Institute of Technology, Shenzhen · Shanghai Jiao Tong University

Abstract

Current test-time scaling (TTS) techniques enhance large language model (LLM) performance by allocating additional computation at inference time, yet they remain insufficient for agentic settings, where actions directly interact with external environments and their effects can be irreversible and costly. We propose ARTIS, Agentic Risk-Aware Test-Time Scaling via Iterative Simulation, a framework that decouples exploration from commitment by enabling test-time exploration through simulated interactions prior to real-world execution. This design allows extending inference-time computation to improve action-level reliability and robustness without incurring environmental risk. We further show that naive LLM-based simulators struggle to capture rare but high-impact failure modes, substantially limiting their effectiveness for agentic decision making. To address this limitation, we introduce a risk-aware tool simulator that emphasizes fidelity on failure-inducing actions via targeted data generation and rebalanced training. Experiments on multi-turn and multi-step agentic benchmarks demonstrate that iterative simulation substantially improves agent reliability, and that risk-aware simulation is essential for consistently realizing these gains across models and tasks.

Figures & tables

Figure 1: Preliminary results on Qwen3-8B. “N” denotes the maximum number of allowed attempts.

Figure 2: The overall framework for ARTIS, where two modes of Iterative Simulation are shown.

Figure 3: Tool simulator building procedure.

Methods	BFCL-v3 Multi-Turn					ACEBench Agent
Methods	Overall	Base	Miss Func	Miss Param	Long Context	End-to-End	Process
Qwen3-8B	12.88	21.00	11.50	9.50	9.50	10.00	14.60
+ Weighted BoN	15.50	24.00	1 6.00	10.50	11.50	15.00	17.60
+ Sequential Revision	16.50	25.50	16.50	1 2.50	11.50	20.00	20.00
+ ARTIS (Parallel)	1 7.75	2 7.50	11.50	16.00	1 6.00	3 5.00	4 3.10
+ ARTIS (Sequential)	18.75	29.50	16.50	1 2.50	16.50	40.00	52.10

Table 1: Main comparison results (accuracy, %) on two benchmarks. Each model is evaluated under two baseline settings (without simulation) and two variants of our method. The best performance in each category within each model is highlighted in bold , while the second best result is u nderlined.

Table 6

Method	Failure Detection				State-Changing Errors
Method	Action-Level Acc. $\uparrow$	Macro Prec. $\uparrow$	Macro Rec. $\uparrow$	Macro F1 $\uparrow$	RAER $\downarrow$	SAER $\downarrow$
Direct	–	–	–	–	39.45	71.52
Sequential Revision	–	–	–	–	31.91	59.76
ARTIS (Normal Training)	9 8.57	8 1.72	8 1.12	8 1.39	2 4.52	5 2.25
ARTIS (Failure-Driven)	99.14	95.15	92.58	93.42	23.76	50.99

Table 4: Failure-specific simulator evaluation and state-changing action errors on the BFCL-v3(%). Failure detection metrics compare simulator predictions with real execution outcomes; RAER and SAER evaluate committed trajectories (lower is better).

Figure 6: Accuracy and evaluation pass rate comparison for different evaluators and summarizers (all Qwen3 models, noted by “Evaluator/Summarizer”). Action agents are 8B.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	N	ACC	API Call Count (#)			Completion Token Count (K#)			Prompt Token Count (K#)
Method	N	(%)	Total	Action	Self-Eval	Total	Action	Self-Eval	Total	Action	Self-Eval
Direct	0	21.0	2497	2497	–	80	80	–	15168	15168	–
	1	23.0	5372	2686	2686	718	87	631	48922	16471	32451
	3	23.5	14588	7475	7113	2339	333	2005	207516	87190	120326
Sequential	5	25.5	26322	13245	13077	3659	494	3165	322449	167564	154885
Revision	8	27.0	39752	20176	19576	5697	814	4883	545918	287325	258592

Appendix

Table 5: Statistics of the baseline Sequential Revision and ARTIS (Sequential) when scaling

N

Methods	BFCL v3
Methods	MT-Base
Qwen3-8B	21.0
+ Perfect Simulator	30.5
+ Self Simulator (Qwen3-8B)	14.0
+ Source Code	22.0
+ Strong Simulator (Qwen3-32B)	22.0

Appendix

Table 6: Results on general-purpose LLMs as simulators with different settings.

Methods	BFCL v3	ACEBench
Methods	MT-Base	Agent Process
Qwen3-8B	21.0	14.6
+ ARTIS (Sequential)	29.5	52.1
- Sum	18.5	24.6
Replace Sum with Majority Voting	27.5	36.4
Replace Sum with Best of N	29.5	28.1

Appendix

Table 7: Results on comparisons of summarization and other choices.

Methods	BFCL v3	ACEBench
Methods	MT-Base	Agent Process
Qwen3-4B	16.5	6.7
+ Sequential Revision	11.0	32.1
+ ARTIS (Sequential)	15.0	30.6
+ Perfect Simulator	24.5	26.7
Llama3.1-8B-Instruct	11.0	6.5

Appendix

Table 8: Results on smaller or less capable backbones, which may fail to realize observable gains.

Methods	BFCL v3 Live					ACEBench Single-Turn
Methods	Overall	Simple	Multiple	Parallel	Para. Multi.	Overall	Single Func	Parallel Func
Qwen3-8B	73.87	77.52	73.22	68.75	6 6.67	62.50	73.00	52.00
+ Sequential Revision	74.09	81.40	72.27	87.50	6 6.67	62.50	68.00	57.00
+ ARTIS (Sequential)	74.32	8 0.62	72.65	8 1.25	75.00	64.00	69.00	59.00
Qwen3-32B	79.57	8 0.62	79.68	8 1.25	62.50	7 0.50	7 5.00	66.00
+ Sequential Revision	80.01	79.46	80.25	8 1.25	75.00	71.50	78.00	6 5.00

Appendix

Table 9: Accuracy (%) comparison on single-turn cases of two benchmarks. The best performance in each category is highlighted in bold , while the second best result is u nderlined.

Figure 7: Thinking vs Non-Thinking.

Seen Tools
Methods	Overall	Base	Miss Func	Miss Param	Long Context
Direct	15.31 $\pm$ 0.83	27.50 $\pm$ 1.25	14.58 $\pm$ 0.72	10.83 $\pm$ 1.44	8.33 $\pm$ 0.72
Sequential Revision	15.94 $\pm$ 0.31	28.33 $\pm$ 0.72	13.75 $\pm$ 1.25	13.33 $\pm$ 3.82	8.33 $\pm$ 0.72
ARTIS (Normal Training)	15.83 $\pm$ 0.65	29.58 $\pm$ 2.60	12.08 $\pm$ 1.44	10.83 $\pm$ 1.91	10.83 $\pm$ 2.60
ARTIS (Failure-Driven)	17.92 $\pm$ 0.79	33.33 $\pm$ 1.44	12.92 $\pm$ 0.72	12.92 $\pm$ 3.82	12.50 $\pm$ 2.17
Unseen Tools

Appendix

Table 10: Accuracy (%) on the strict BFCL seen/unseen tool split with Qwen3-8B as the action agent. Results are “mean

\pm

std” over three runs. Best means within each split are in bold .

Method	Accuracy $\uparrow$					Compute per Case $\downarrow$
Method	Overall	Base	Miss Func	Miss Param	Long Context	LM Calls	Simulated Tool Steps	Tokens
WMA-style	15.57	21.05	13.16	12.28	15.79	196.0	83.1	1.30M
SVS-style	15.03	20.18	11.84	11.40	16.67	192.0	84.6	1.27M
ARTIS	17.00	22.81	12.72	12.72	19.74	115.7	65.3	0.96M

Appendix

Table 11: Matched simulator-assisted baseline result comparisons (Accuracy (%)) on the BFCL-v3 unseen-tool split using Qwen3-8B and the same failure-driven simulator. Compute is reported per case across all four categories. Tokens include input and output (M: million). Best means are in bold .

Explore similar work

No Time Like the Present: Agentic Test-Time Training for LLM Agents
Jul 3, 2026Yanbo Wang, Jinhua Hao, Yuze Shi +2Continual Learning for LLM Agents Test-Time Training
Kuaishou Technology
LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling
May 8, 2026Tong Zheng, Haolin Liu, Chengsong Huang +10LLM Inference Efficiency Automated Algorithm Discovery
UMD · UVA · WUSTL +3
Evaluating Test-Time Scaling of General LLM Agents
Feb 22, 2026Xiaochuan Li, Ryan Ming, Pranav Setlur +6LLM Agent Evaluation Test-Time Scaling
Language Technologies Institute, School of Computer Science, Carnegie Mellon University · Meta

Jul 3, 2026J/K move · Enter open · S save

No Time Like the Present: Agentic Test-Time Training for LLM Agents

Yanbo Wang, Jinhua Hao, Yuze Shi, Kun Yuan+1

Kuaishou Technology

Continual Learning for LLM Agents Test-Time Training Long-Horizon LLM Agents

Open paper AskPDF arXiv

LLM agents often degrade over long episodes: as trajectories grow, they revisit explored states, repeat failed actions, and lose strategies that previously worked. Test-time training (TTT) offers a way to adapt model weights to the evolving task state, but existing LLM TTT methods largely adapt once to a fixed input. We study continuous TTT in multi-turn agent episodes, where each update changes the policy that generates later training text. This creates a self-training loop that helps when new trajectory information appears, but can amplify drift when the agent gets stuck and repeatedly trains on similar text. We find that update-text repetition distinguishes these regimes and introduce Agentic Test-Time Training (aTTT), a token-level reweighting method that downweights the loss on tokens appearing in repeated

n

-grams from prior updates while leaving novel tokens fully weighted. To run such updates inside live episodes, we build a concurrent serving system using vLLM's runtime LoRA API, limiting overhead to 1.9

\times

the no-TTT cost. aTTT improves success by up to 5.0 points on ALFWorld and 4.9 points on SWE-bench Lite. The gains concentrate where models already have task competence but drift over long trajectories, suggesting that aTTT mainly preserves existing competence rather than teaching new abilities.

Abstract

Explore similar work

No Time Like the Present: Agentic Test-Time Training for LLM Agents

LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

Evaluating Test-Time Scaling of General LLM Agents