LLM agents increasingly solve long-horizon tasks by autonomously interacting with their environment. In doing so, their strategies rely on assumptions about that environment: which resources and tools exist, where they are located, and how they behave. When these assumptions no longer hold, reliable agents must detect the change and adapt while pursuing the same goal. We study this adaptation capability through environmental novelty: a change that keeps the task objective fixed while invalidating an assumption underlying an otherwise successful trajectory. We introduce AGNI, an automated pipeline that extracts trajectory-relevant assumptions, injects targeted environmental changes, and validates that the resulting novel tasks remain solvable. Across three terminal benchmarks, AGNI produces diverse novelties spanning resources, interfaces, constraints, and execution semantics. Evaluating multiple LLM agents reveals a substantial adaptation gap between base and novel tasks. Trajectory analysis suggests that agents often encounter evidence of the change but fail to diagnose its cause and revise their strategy. Finally, post-training for environmental novelty improves adaptation to held-out novel tasks while also improving performance on base tasks. Our results highlight a gap between task competence and adaptive capability and motivate environmental variation as a core dimension of agent training and evaluation.
Figures & tables
Figure 1: Adaptation to Environmental Novelty in Terminal Agents. (a) Novelty can disrupt an agent’s original strategy and recovery depends on adaptive behaviors during interaction (example from Appendix C.7 ). (b) AGNI injects novelties into base environments such that task success requires adaptation, while preserving task objective and solvability. (c) We use AGNI -generated tasks to measure adaptation, analyze adaptive behaviors, and post-train agents to improve adaptation.
ET-eval
TB-Lite
TB-2
Model
Base
Novel
Δadapt
Base
Novel
Δadapt
Base
Novel
Δadapt
GPT-5.5
97.0
74.5
22.5
93.2
71.7
21.5
88.8
79.7
9.1
GPT-5.4-mini
94.5
61.9
32.6
65.0
39.5
25.5
75.9
46.6
29.3
Kimi-2.6
92.7
74.7
18.0
83.6
58.3
25.3
73.8
61.4
12.4
DeepSeek-V4-Flash
89.4
74.2
15.1
70.2
49.3
20.8
66.0
54.0
12.0
GPT-OSS-120B
78.6
29.4
49.2
52.7
17.6
35.1
28.0
13.3
14.7
Table 1: Environmental novelty consistently degrades pass@1 (%) across models and datasets. Moreover, models with stronger base performance do not always exhibit smaller adaptation gaps. Here, Δadapt denotes the Base-Novel pass@1 gap in percentage points.
Model
Effort
Base
Novel
Δadapt
GPT-5.4-mini
Low
85.4
41.5
44.0
Medium
91.4
53.8
37.6
High
94.5
61.9
32.6
GPT-OSS-120B
Low
75.0
22.2
52.8
Medium
78.6
29.4
49.2
High
84.2
35.3
48.9
Table 2: Pass@1 (%) on ET-eval across reasoning efforts. Higher effort reduces Δadapt for GPT-5.4-mini and GPT-OSS-120B but has a non-monotonic effect for Kimi-2.6.
Figure 2: Novelty disclosure substantially recovers the adaptation gap on ET-eval. Bars show pass@1 for base and novel tasks, with disclosure gain Δdisc ; labels report corresponding recovery fraction ρrec .
Figure 3: Interaction behaviors associated with adaptation on ET-eval. (a) Better adapting models show larger increases in interaction length when going from base to novel tasks; lines mark medians, diamonds means, and labels report mean novel-base difference. (b) Better adapting models show more reliable progress from recognizing novelty to diagnosing it and revising their strategy.
Figure 4: Generic post-training raises absolute performance but does not close the adaptation gap. Bars show base and novel pass@1; ET-eval is solid and TB-Lite dotted. Δ reports the adaptation gap.
Figure 5: Novelty-augmented training narrows the ET-eval adaptation gap and transfers to TB-Lite. Bars show base and novel pass@1; ET-eval is solid and TB-Lite dotted. Δ reports the adaptation gap.
Method
Base
Novel
Δadapt
GRPO on Dnovel-paired
59.82
29.46
30.36
+ Meta Prompt
61.25
32.59
28.66
+ Meta Prompt Training
64.29
37.86
26.43
+ Recovery & Verify
69.55
38.00
31.55
+ Diagnosed Exploration
53.93
25.14
28.79
+ Exploration & Completion
66.96
38.80
28.16
Table 3: Meta-action reasoning improves adaptation; process rewards provide limited additional gains. Table reports ET-eval pass@1 (%) and adaptation gap.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Role
Symbol
Purpose
Solver
MS
Generates interaction trajectories in the base and novel environments.
Strategy analyzer
MA
Summarizes observed task-solving behavior and extracts trajectory-grounded environmental assumptions.
Novelty generator
MNdim,MNspec
Proposes environmental changes that target extracted assumptions while preserving the task objective.
Implementer
MNimpl
Instantiates a novelty specification as a modified environment E′ and produces an adapted reference solution S′ .
Behavioral observer
MAobs
Checks whether the injected novelty produces the expected change in behavior between the base and novel environments.
Trajectory analyzer
Analyze Trajectories
Analyzes novel-task trajectories to characterize adaptation and identify unintended shortcuts or reward hacking.
Appendix
Table 4: LLM roles in AGNI .
Setting
Value
Model
GPT-5.5 (2026-04-24), all roles.
Reasoning effort
High for the solver; medium for other roles.
Sampling
Temperature and top- p unset (API defaults); no fixed seed.
Output-token limits
8,192 per solver turn; 32,000 per non-solver call.
Solver harness
Terminus-2 (Harbor 0.6.6); at most 32 turns.
Construction rollouts
K=8 per base and novel task, across all three sources.
Appendix
Table 5: AGNI model settings and construction budgets.
Axis
Strata
Base-task difficulty
Easy, medium, hard
Novelty-induced difficulty
Low, medium, high
Occurrence plausibility
Low, medium, high
Realization fidelity
Low, medium, high
Mechanism family
Categorical labels from the frozen taxonomy
Appendix
Table 6: Strata used for joint sampling across ET-eval, TB-Lite, and TB-2.
Adaptive behavior
Non-adaptive behavior
Agent
GPT-5.4-mini, run_02 .
GPT-5.4-mini, run_07 .
Explanation
Reads the launch log and identifies nohup as the obstacle.
Leaves the cause unresolved and continues to rely on nohup .
Action
Relaunches without nohup , using python3 -m mlflow server … & .
Retries launch commands that still use nohup .
Check
Checks the server’s health endpoint and confirms that it responds.
Launch attempts continue to fail; no working server is established.
Result
Trains and registers the model; all three tests pass.
Does not complete the task; all three tests fail.
Appendix
Table 7: Adaptive and non-adaptive responses to a blocked launch command.
Adaptive behavior
Non-adaptive behavior
Agent
GPT-5.4-mini, run_00 .
GPT-5.5, run_02 .
Explanation
Identifies the changed encoding as the reason direct searches miss the text.
Concludes that the logs contain none of the target words.
Action
Uses Python to decode the logs as UTF-16 before counting matching lines.
Repeats grep searches without accounting for the encoding.
Check
Confirms the UTF-16 encoding marker in the file bytes.
Accepts repeated zeros as evidence that the target words are absent.
Result
Reports counts of 4, 3, and 8; all three tests pass.
Reports zero counts; the grader rejects the summary.
Appendix
Table 8: Adaptive and non-adaptive responses to changed log encoding.
Adaptive behavior
Agent
GPT-5.4-mini, run_00 .
Explanation
Suspects an incorrect program path in the first line or Windows line endings; does not directly inspect the extra character.
Action
Reads the script and runs bash /app/generate_zk_protocols.sh , avoiding the broken first line.
Check
Confirms that the script exists and has execution permission before changing how it is run.
Result
Runs the generator and writes both reports; all 15 tests pass.
Appendix
Table 9: An adaptive response to a broken script launch.
Adaptive behavior
Non-adaptive behavior
Agent
GPT-OSS-120B, run_00 .
GPT-OSS-120B, run_01 .
Explanation
Finds that a configuration setting prevents pip from searching for packages.
Attributes the remaining failure to Python-version compatibility without confirming that explanation.
Action
Inspects pip configuration and removes no-index from /etc/pip.conf .
Restores pip but leaves its global configuration unchecked.
Check
Successfully installs requests after removing the setting.
Confirms pip’s version but accepts No matching distribution found when testing installation.
Result
Restores package installation; both tests pass.
Leaves installation broken and declares completion. Verifier setup also fails, so the task tests do not run.
Appendix
Table 10: Adaptive and non-adaptive responses to a package-installation restriction.
Adaptive behavior
Non-adaptive behavior
Agent
GPT-5.4-mini (high), run_01 .
GPT-5.4-mini (high), run_00 .
Explanation
Suspects that an inherited TAR_OPTIONS setting changes the stored names.
Recognizes the wrong names but does not identify the setting that changes them.
Action
Unsets TAR_OPTIONS when recreating and extracting the archive.
Repeats the archive command without changing the setting.
Check
Lists the rebuilt archive and verifies the names app.log and error.log .
Sees the raw/ prefix again but stops after extraction fails.
Result
Extracts the log and completes the outputs; all four tests pass.
Leaves the archive incorrect and the extracted log missing; three tests fail.
Appendix
Table 11: Adaptive and non-adaptive responses to changed archive filenames.
Adaptive behavior
Non-adaptive behavior
Agent
GPT-OSS-120B, run_00 .
GPT-OSS-120B, run_02 .
Explanation
Identifies that the script treated a repeated header as a release.
Blames a blank line at the end of the CSV for creating an extra file.
Action
Reads the CSV, removes the generated files, and rebuilds the outputs while skipping header lines.
Does not read the CSV. Attempts to delete release-.json , although the extra file is named release-version.json .
Check
Lists the regenerated files and confirms that exactly three remain.
Sees that the count is still four but declares the task complete.
Result
Completes the task; all seven tests pass.
Leaves the extra file and log entry; two tests fail and five pass.
Appendix
Table 12: Adaptive and non-adaptive responses to a repeated CSV header.
Adaptive behavior
Non-adaptive behavior
Agent
GPT-5.5, run_01 .
GPT-5.5, run_03 .
Explanation
Reads the startup error and identifies an environment setting that requests a missing database script.
Treats a successful build and the expected JAR path as sufficient to declare completion.
Action
Sets the database connection explicitly in the application and rebuilds the JAR.
Configures the database through the properties file, which the environment setting overrides.
Check
Launches the rebuilt JAR, confirms startup, and tests login.
Checks that the JAR exists, but never launches it or tests an endpoint.
Result
Runs the service successfully; all ten tests pass.
The grader cannot start the service; all ten tests report setup errors.
Appendix
Table 13: Adaptive and non-adaptive responses to an overridden database setting.
Adaptive behavior
Non-adaptive behavior
Agent
Kimi-2.6, run_03 .
Kimi-2.6, run_04 .
Explanation
Recognizes that Git’s saved decision has left the requested biography update missing.
Recognizes that Git kept the old biography, but treats the conflict as resolved.
Action
Retrieves the updated biography from the project history and saves it.
Saves the result with the old biography still in place.
Check
Reads the biography and checks whether it includes the requested Stanford update.
Reads the old biography but accepts Git’s resolved status as sufficient.
Result
Restores the updated biography and page layout; both tests pass.
Restores the layout but leaves the biography unchanged; one test fails and one passes.
Appendix
Table 14: Adaptive and non-adaptive responses to Git reusing an old decision.
Score
Occurrence plausibility
Realization fidelity
1
Implausible or specific to the evaluator
Unfaithful or incoherent implementation
2
Technically possible but contrived
Crude proxy with major artificial artifacts
3
Plausible in some deployments
Controlled abstraction preserving the central effect
4
Recognizable real deployment or configuration change
Faithful simulation with minor simplifications
5
Documented, observed, or widely recognized failure mode supported by the artifacts
Direct or near-exact reproduction of the real mechanism
Appendix
Table 15: The two realism scores are assigned separately using this rubric.
Occurrence plausibility
Realization fidelity
Dataset
Pairs
Counts (1–5)
Mean
≥4
Counts (1–5)
Mean
≥4
ET-eval
140
0, 2, 43, 86, 9
3.73
68%
0, 0, 20, 35, 85
4.46
86%
TB-Lite
118
0, 6, 25, 76, 11
3.78
74%
0, 1, 11, 41, 65
4.44
90%
TB-2
87
0, 2, 18, 60, 7
3.83
77%
0, 0, 12, 29, 46
4.39
86%
All
345
0, 10, 86, 222, 27
3.77
72%
0, 1, 43, 105, 196
4.44
87%
Appendix
Table 16: LLM realism ratings for the final evaluation sets. Counts list scores 1–5; ≥4 is the percentage rated 4 or 5, rounded to the nearest integer. Median plausibility and fidelity are 4 and 5, respectively, in every set.
Model
Provider
Model / snapshot
Main effort
Other efforts
GPT-5.5
OpenAI
gpt-5.5 (2026-04-24)
High
–
GPT-5.4-mini
OpenAI
gpt-5.4-mini (2026-03-17)
High
Medium, Low
GPT-OSS-120B
OpenAI
gpt-oss-120B
Default
High, Low
Kimi-2.6
Moonshot
Kimi-K2.6 (2026-04-20)
Default
High, Low, None
DeepSeek-V4-Flash
DeepSeek
DeepSeek-V4-Flash-0731 (2026-07-31)
None
–
Grok-4.20
xAI
grok-4.20-beta-0309
Non-reasoning
–
Appendix
Table 17: Evaluation models. Default means reasoning_effort was omitted (approximately medium for GPT-OSS-120B and Kimi-2.6). Other efforts are used in Table 2 .
Setting
Value
Agent harness
Terminus-2, at most 32 interaction turns.
Rollouts
K=5 per model–task pair, unless otherwise specified.
Completion budget
8,192 tokens per model response.
Sampling
Temperature and top- p unset (deployment defaults); no explicit seed.
Verifier timeout
10× the configured verifier timeout.
Appendix
Table 18: Shared settings for the main evaluation.
ET-eval
TB-Lite
TB-2
Interact@1
Interact@1
Interact@1
Model
Base
Novel
Bypass@1
Fail w/o Int.
Base
Novel
Bypass@1
Fail w/o Int.
Base
Novel
Bypass@1
Fail w/o Int.
GPT-5.5
99.1
99.0
0.7
1.2
96.9
96.3
1.7
7.2
96.6
98.3
1.4
1.4
GPT-5.4-mini
95.3
95.9
2.7
3.8
92.7
94.9
1.5
8.6
91.2
95.0
0.7
8.1
Kimi-2.6
97.3
98.4
1.6
0.0
94.2
97.8
0.8
3.3
92.6
96.1
2.3
4.2
DeepSeek-V4-Flash
96.1
97.8
1.3
3.4
95.4
96.6
0.7
5.4
94.6
95.5
0.8
4.2
Appendix
Table 19: Cross-model relevance of curated novelties (%).
Retry prevalence
Retry persistence
Model
Rollouts with ≥1 Retry (%)
P(Retryt+1∣Retryt) (%)
GPT-5.5
9.8
9
GPT-5.4-mini
15.2
50
Kimi-2.6
18.1
6
DeepSeek-V4-Flash
17.4
10
GPT-OSS-120B
27.8
40
Appendix
Table 20: Retry prevalence and persistence on ET-eval novel rollouts (%).
Stage
Definition
Interacted
An executed command touches an affected tool, path, or file. Detected deterministically; no visible effect is required.
Observed
Terminal output exposes novelty-related evidence, such as an error, unexpected result, or changed file state. Recognition is not required.
Recognized
The agent explicitly identifies the observation as unexpected or inconsistent with its assumptions, without necessarily identifying the cause.
Diagnosed
The agent identifies the changed condition sufficiently to explain the problem and guide a response. Restating a symptom is insufficient.
Revised
A changed strategy successfully removes or bypasses the obstacle while preserving the task. Failed attempts and exploits do not count.
Solved
The agent completes the original task under novelty using a task-preserving strategy.
Appendix
Table 21: Stages of novelty handling used in trajectory analysis.
Model
Diag. → Rev.
Diag. → Retry
Premature Done
Pass ∣ Diag.
Pass ∣ Rev.+Diag.
Pass ∣ Rev. w/o Diag.
GPT-5.5
71
5
14
96
96
68
Kimi-2.6
41
4
11
86
87
50
DeepSeek-V4-Flash
47
5
12
86
86
70
GPT-5.4-mini
57
11
20
87
86
59
GPT-OSS-120B
30
10
21
65
64
22
Grok-4.20
31
14
16
61
64
20
Appendix
Table 22: Detailed behavioral transitions and conditional success rates on ET-eval. Diag. → Rev. and Diag. → Retry measure revision or retry immediately after diagnosis. Premature Done denotes trajectories that finalize immediately after a failed or partial step. The final columns report success conditioned on diagnosis and on whether strategy revision occurs with or without a preceding diagnosis. All values are percentages.
Setting
Value used in our experiments
Optimizer
AdamW; β1=0.9 , β2=0.95
Weight decay
0.01
Learning rate
10−6 , constant; no warmup
Gradient clipping
0.2
Batch size
16
Rollouts per prompt
16
Appendix
Table 23: Training hyperparameters used in our experiments, adopted from Shrivastava et al. (2026) . ECHO-specific settings apply only to ECHO runs; run lengths are stated above.
ET-eval
TB-Lite
Qwen3-14B GRPO
Base
Novel
Base
Novel
Clean
65.3
24.6
17.2
2.9
Novel-paired
67.1
35.5
18.9
5.5
Appendix
Table 24: Qwen3-14B GRPO pass@1 (%) on ET-eval and TB-Lite.
Figure 6: Process rewards change behavior but yield limited gains on novel tasks. Held-out ET-eval, 1,120 novel rollouts per arm. Failed commands are a percentage of calls; Table 3 reports task success.
Figure 7: Qwen3-14B generic post-training on ET-eval (solid) and TB-Lite (dotted). Bars show base and novel pass@1 (%); Δ marks the adaptation gap.
Prompt
Purpose
Usage in Algorithm 1
Defines novelty for reuse across prompts
Lines 3, 4, 6, 11, 16: shared model context
Characterizes the base strategy and assumptions
Line 3: MA
Identifies applicable novelty dimensions
Line 4: MNdim
Synthesizes novelty specifications
Line 6: MNspec
Instantiates Ec′ and adapted Sc′
Line 11: MNimpl
Repairs a failed task instantiation
Line 11: MNimpl retry (implicit)
Appendix
Table 25: Prompt-to-operation mapping for the AGNI pipeline and disclosed-novelty evaluation. Line numbers refer to Algorithm 1 .
LLM-based agents are assumed to integrate environmental observations into their reasoning: discovering highly relevant but unexpected information should naturally lead to a model exploiting its own discoveries. We show that this assumption is false for current LLM-based agents, which struggle to reflect or react to unexpected information. Across three benchmarks (Terminal-Bench, SWE-Bench, AppWorld), we inject complete task solutions into the agent environments to deliberately expose a task's solution to a model. While agents discover these solutions on Terminal-Bench in 79-81% of runs, they interact, or exploit, them in only 37-50% of cases. This gap is starkest in AppWorld: agents see documentation stating that a command "returns the complete solution to this task" in over 90% of attempts but exploit this in fewer than 7% of trials. We show that agents lack what we call environmental curiosity: the capability to recognize and investigate unexpected but relevant observations in response to environmental stimuli. We identify three main factors influencing environmental curiosity: available tools in the agent scaffold, test-time compute, and training data distribution. Our findings identify configurations that maximize curiosity also achieve the best performance on the unmodified benchmarks. Yet even jointly optimized agents still ignore discovered solutions in the majority of trials: current agents use the environment to fetch expected information, but not to revise their strategy or maximally exploit useful stimuli.
Many real-world tasks (e.g., office workflows, scientific experimentation) require LLM agents to interact repeatedly with their environments for context-dependent operations. However, such environments are often not agent-ready. First, information is often scattered and fragmented across the environment. Second, relevant evidence in the environment is often mixed with misleading information and conflicting versions. Third, environments evolve over time, introducing new noise and more challenging tasks. These challenges can substantially degrade performance for state-of-the-art AI agents (e.g., from 83.9% to 57.6%). To address these challenges, we propose Env-Rethink (a system with 27B post-trained model) that supports three main capabilities: (1) It adaptively builds Collection Maps (for organizing related files) and Event Logs (for contextualizing cross-data relationships) to supplement necessary context; (2) It further leverages the post-trained model (through offline trajectory learning) to identify underlying noise issues in the environment; (3) It ultimately evolves environments through virtual event histories that alter environmental states and evidence relationships, producing more tricky ones for further agent improvement. Experiments show that Env-Rethink can effectively improve downstream task performance (with a 15.1 percentage-point increase in mean rubric pass rate across nine models on 30 tasks).
Yukai Wu, Yuanjing Yang, Le Zhou +8
Shanghai Jiao Tong University · Theseus Labs · Tencent Hunyuan
Large language model (LLM) agents increasingly rely on reusable external skills to solve long-horizon interactive tasks. Existing training-free skill adaptation pipelines usually update skills from full trajectories or session-level feedback, which makes failure attribution coarse and often produces unstable or overly broad revisions. We propose SkillAdaptor, a training-free step-level skill adaptation framework with explicit failure attribution, and it can plug into OpenClaw-class agent harnesses. Given a failed trajectory, SkillAdaptor identifies a first actionable fault step, links responsibility to candidate skills, and applies targeted updates under explicit acceptance checks while keeping the backbone frozen. We evaluate on WebShop, PinchBench, and Claw-Eval with Kimi-K2.5, GLM-5, and GPT-5.2. SkillAdaptor improves over no-skill and skill-adaptation baselines on all three suites, with the largest single-metric improvements of +1.5 points on PinchBench Avg Score%, +1.8 on Claw-Eval Avg Score, and +1.7 on WebShop success rate. These results indicate that step-level attribution supports more stable and auditable training-free skill maintenance\footnote{The code will be released at https://github.com/zjunlp/SkillAdaptor.}.
Zhuoyun Yu, Xin Xie, Wuguannan Yao +4
♠Zhejiang University · ♣Ant Digital Technologies, Ant Group · ♢Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph +1