Learning Simple Test-Time Environments for LLM Web Agents
Authors: Junxuan Li, Zijun Liu, Ziyi Huang, Peng Li, Yuzhou Liu, Ming Yan, Yang Liu
Organizations: College of Computer Science and Technology, Jilin University, Jilin, China · Dept. of Comp. Sci. & Tech., Tsinghua University, Beijing, China · School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Tongyi Lab, Alibaba Group
Large language model (LLM) agents have demonstrated remarkable proficiency in manually constructed environments, yet their performance frequently collapses when transitioned to complex real-world settings. Existing research largely attribute this degradation to the compositional generalization gaps in LLMs on combinations of multiple simple, well-structured environments. In this work, we propose that LLM web agents can learn simple environment observations at test time. Specifically, we introduce trial steps for agents to decompose a complex environment observation into sub-modules, and implement a label-free learning method, Test-Time Environment Decomposition (TTED), to adapt agent behaviors with experience during inference. Our empirical evaluations demonstrate the framework's efficacy across both synthetic and realistic benchmarks, showing (1) experience gains acquired within simpler sub-environments can be effectively composed to improve performance in the full one, and (2) test-time training on sub-environments can significantly enhance the compositional generalization of agents in real-world web automation tasks. We also provide key insights in the design of the label-free learning algorithm. As more complex environments are accessed by LLM agents, we believe learning environment decomposition skills at test time will be critical for robust real-world deployment.
Figures & tables
Figure 1: An overview of (a) environment decomposition and a performance comparison of (b) test-time training (TTT) and TTED. “SR” denotes success rate. “GT” denotes ground truth. (a) In complex real-world tasks, e.g., writing academic papers, the environment is often compositionally complex. Decomposing into simple test-time environments can help the agent explore and self-assess more effectively, resulting in improved performance after test-time learning. (b) TTED recovers performance degradation of TTT and outperforms RL on full ground-truth rewards. *We only use a subset of WebArena for evaluation.
Figure 2: Task success rates (%) on CompWoB+ with Qwen3-8B (left) and GPT-4o-mini (right). TTED effectively mitigates compositional generalization failures on both models.
Figure 3: The Test-Time Environment Decomposition (TTED) methodology, illustrating the two-stage process of (1) Trial Steps with environment decomposition and (2) Test-Time Learning and Execution, utilizing either ICL or RL. The agent gathers experience within the decomposed sub-environments through trials. And ICL leverages correct past actions as task-specific demonstrations to solve tasks within single-turn, static webpages, while RL optimizes both decomposition process and action generation policy with trajectories from multiple environments, enabling adaptation in multi-turn, realistic web environments.
Figure 4
Method
Reward Labeling
#Steps
Reward Consistency
rgoal
robs
raction
Overall
TTRL
self-consistency
4.05
-
-
0.68
0.68
TTT
principled GRM
7.18
-
-
0.71
0.71
TTED
principled GRM
8.23
0.81
0.87
0.86
0.85
+ TTRL (action)
self-consistency
8.44
-
-
0.80
0.80
Table 2: Consistency evaluation (normalized AD) of reward labeling before model training from test-time learning methods with humans. Evaluation is on 82 tasks from WebArena.
Advantage Estimation
#Sub-Envs
Train. w/ GT
TTT
TTED
SFT
3200
14.1
11.2
17.6
REINFORCE w/ Advantage Normalization
800
-
12.5
19.4
1600
-
13.5
18.4
2400
-
13.2
20.0
3200
14.8
12.5
21.0
REINFORCE
3200
13.0
14.3
17.6
Table 3: Test-time scaling experiments for test-time learning methods on WebArena with different advantage estimation strategies. We report task success rates (%) under different training budgets, i.e. numbers of explored sub-environments.
Method
WebArena
TTED
21.0
- Sub-goal decomposition
12.4
- Sub-observation decomposition
15.0
- Both = TTT
12.5
Table 4: Ablation of TTED components on WebArena, by removing sub-goal and sub-observation decomposition. We report task success rates (%) under the same RL budget.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Task success rates (%) on CompWob+ with Qwen3 14B
Model
Method
WebArena SR
Gemma3-12B
TTI
22.1
GPT-4o
-
23.5
GPT-5.5*
-
67.3
Qwen3-8B
-
12.1
RL w/ GT
14.8
TTT (RL)
12.5
Appendix
Table 5: Success rate (SR, %) on the selected WebArena subset. Results in gray are provided as contextual references only. TTI is not a test-time training method and uses ground-truth labels in the benchmark. GPT-5.5* is evaluated on full WebArena-Verified ( OpenAI 2026b ) , which shares similar task difficulty but uses different verifiers.
Method
#Sub-Envs
WebArena SR (%)
#Steps
#Prompt Tokens
Qwen3-8B
0
12.1
4.72
6338.9
TTED (RL)
3200
21.0
8.29
3948.2
Appendix
Table 6: Efficiency Measures of TTED-trained models on WebArena. “#Steps” denotes the average number of interaction steps in a task, and “#Prompt Tokens” denotes the average input tokens of the action generation policy.
Figure 6: Observation for # Case 1.
Figure 7: Observation for # Case 2.
Figure 8: Observation for # Case 3.
Figure 9: Observation for # Compwob+.
Figure 10: Overall workflow of TTED with in-context learning (ICL) on CompWoB+. Given a complex goal and the original observation, the agent first performs environment decomposition to divide the task into multiple sub-goals and extract the corresponding sub-observations. For each decomposed sub-environment, the agent generates trial actions and evaluates them through the self-assessment module. The successful trials are stored as experiences, while failed trials trigger retry until success or reaching the maximum retry limit. Finally, the collected successful experiences are leveraged as demonstrations for ICL, guiding the agent to directly generate the final action sequence in the original environment.
Figure 11: Workflow of the self-assessment module in TTED with ICL. For each generated action, a simulated environment is first constructed from the HTML snippet using Selenium. The action is then executed within the simulated environment, and the resulting webpage state is analyzed. A JavaScript verification script is generated to inspect whether the action is performed correctly. The execution feedback, observations, and verification results are integrated to determine whether the action is successful and returns the final judgment.
ID
Base Task
ID
Base Task
1
click_button
6
login_user
2
click_link
7
choose_list
3
click_widget
8
read_table
4
click_checkboxes
9
edit_text
5
click_option
10
enter_time
Appendix
Table 7: Base tasks used for constructing composite tasks in CompWoB+.
Composition Size
Task
Instances / Task
Total Instances
3
10
10
100
4
10
10
100
5
10
10
100
6
10
10
100
7
10
10
100
8
10
10
100
Appendix
Table 8: Statistics of composite tasks in CompWoB+.
CMS
Map
Shopping
Gitlab
Reddit*
Overall
#Tasks
149
124
174
114
0
561
Appendix
Table 9: Statistics of the selected WebArena task subset. *For Reddit, we remove all tasks due to network issues.
Adv. Estimation
Method
Inf. Time (s)
#Tokens
#Steps
GPU h.
SFT
SFT w/ GT
–
–
–
2.24
TTT
- -
–
–
11.07
TTED
- -
–
–
12.53
REINFORCE
RL w/ GT
–
–
–
8.67
TTT
0 16.7
17514.9
6.17
14.40
TTED
0 46.8
26114.5
9.91
14.80
Appendix
Table 10: Computational costs for different adaptation methods on WebArena. All methods are evaluated under a uniform training budget of 3200 sub-environments. GPU hours are estimated with Qwen3-8B on 8 NVIDIA A100-SXM4-80GB GPUs. Avg. inference time and token numbers are measured per sub-environment exploration.