Learning Situation-Conditioned Thinking Policies for Long-Term LLM Agents

Hong Su

cs.AIOct 7, 2026

Learning Situation-Conditioned Thinking Policies for Long-Term LLM Agents

Authors: Hong Su

Organizations: School of Computer Science, Chengdu University of Information Technology, Chengdu, China

Abstract

Long-running autonomous agents must reuse accumulated reasoning experience without allowing explicit historical memory and LLM context to grow indefinitely. However, existing memory mechanisms mainly retrieve, summarize, or compress past content and do not directly learn when particular kinds of thinking should be activated or discover new thinking knowledge from temporally dispersed experiences. This paper proposes a situation-conditioned thinking memory framework that transforms historical reasoning experience into a lightweight policy for predicting what should be thought about in the current situation, while leaving detailed reasoning to a large language model. Situations may represent temporal or spatiotemporal evolution rather than only current states. Temporary experiences are also periodically analyzed across multiple independent episodes to identify repeated long-range regularities, which are consolidated into new thinking knowledge and further internalized by the lightweight policy. Experiments show that the learned policy achieves 1.000 F1 on temporal-rule generalization, improves DeepSeek reasoning F1 from 0.789 to 0.868, reduces online processing time from 0.3636 ms to 0.0382 ms per query at 30,000 historical situations, and reaches 1.000 relation-discovery F1 and future-thinking accuracy after sufficient repeated cross-experience evidence.

Figures & tables

History $N$	Summary-RAG F1	Sequence-RAG F1	Seq.-RAG MB	Seq.-RAG ms/query	Proposed F1	Proposed MB	Proposed ms/query
500	$0.528\pm 0.068$	$1.000\pm 0.000$	0.156	0.0186	$1.000\pm 0.000$	7.358	0.0382
2,000	$0.518\pm 0.042$	$1.000\pm 0.000$	0.624	0.0309	$1.000\pm 0.000$	7.358	0.0382
10,000	$0.499\pm 0.073$	$1.000\pm 0.000$	3.120	0.1533	$1.000\pm 0.000$	7.358	0.0382
30,000	$0.495\pm 0.071$	$1.000\pm 0.000$	9.360	0.3636	$1.000\pm 0.000$	7.358	0.0382

TABLE I: Long-history scalability with top-

4

retrieval. Storage uses decimal MB (

10^{6}

bytes).

Fig. 1: Storage as explicit historical experience grows. Sequence-RAG grows with the number of stored histories, while the consolidated thinking policy remains fixed.

Fig. 2: Online cost under growing history size. Sequence-RAG must search a larger explicit memory, whereas the learned policy performs fixed-size inference.

Experiment	Method	Micro-F1
Exp. 2	Proposed Transformer	$1.000\pm 0.000$
	Summary-RAG	$0.566\pm 0.045$
	RF-Summary	$0.383\pm 0.033$
Exp. 3	Proposed Sequence	$1.000\pm 0.000$
	Current-State-Only	$0.711\pm 0.044$

TABLE II: Mechanism verification for temporal-order generalization and situation evolution.

Method	Factor Precision	Factor Recall	Factor F1	Decision Accuracy	Avg. Tokens
DeepSeek-Only	$0.704\pm 0.055$	$1.000\pm 0.000$	$0.789\pm 0.038$	$0.484\pm 0.115$	$719.61\pm 0.98$
RAG+DeepSeek	$0.835\pm 0.039$	$1.000\pm 0.000$	$0.877\pm 0.029$	$0.559\pm 0.106$	$727.94\pm 0.82$
Proposed+DeepSeek	$0.815\pm 0.048$	$1.000\pm 0.000$	$0.868\pm 0.035$	$0.575\pm 0.140$	$728.13\pm 1.18$
Oracle+DeepSeek	$0.835\pm 0.039$	$1.000\pm 0.000$	$0.877\pm 0.029$	$0.559\pm 0.106$	$727.94\pm 0.82$

TABLE III: End-to-end reasoning with deepseek-v4-flash . Values are mean

\pm

standard deviation over five seeds.

Comparison / Metric	Mean Diff.	95% CI	$p$
Proposed – DeepSeek / F1	$+0.0786$	$[0.0609,0.0974]$	$<0.001$
Proposed – DeepSeek / Decision	$+0.0906$	$[0.0594,0.1250]$	$<0.001$
Proposed – RAG / F1	$-0.0089$	$[-0.0302,0.0135]$	$0.472$
Proposed – RAG / Decision	$+0.0156$	$[-0.0281,0.0594]$	$0.587$

TABLE IV: Paired per-situation analysis in Experiment 4 (

n=320

Fig. 3: Factor F1 in the end-to-end DeepSeek experiment. Learned thinking guidance improves over unguided DeepSeek and approaches the explicit-retrieval and oracle references.

Historical situations	Proposed micro-F1
100	$0.777\pm 0.077$
250	$0.912\pm 0.051$
500	$0.996\pm 0.005$
1,000	$0.995\pm 0.006$
2,000	$0.998\pm 0.003$
4,000	$0.999\pm 0.001$

TABLE V: Rare-pattern learning as historical experience accumulates.

Fig. 4: Learning of a rare thinking pattern as the number of historical situations increases. Error bars show one standard deviation over five seeds.

At 8 repetitions, No-Consolidation, Unordered, and Local relation F1 are all $0.000\pm 0.000$ .
Repetitions/rule	Candidate Recall	Statistics-Only F1	Proposed+DeepSeek F1	Oracle F1
1	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$	$1.000\pm 0.000$
2	$0.000\pm 0.000$	$0.000\pm 0.000$	$0.000\pm 0.000$	$1.000\pm 0.000$
4	$0.933\pm 0.149$	$0.333\pm 0.471$	$0.333\pm 0.471$	$1.000\pm 0.000$
8	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$

TABLE VI: Cross-experience knowledge discovery. Candidate recall and relation F1 are reported over five seeds. Future-thinking accuracy is evaluated at eight repetitions.

Knowledge source	Future-thinking accuracy
No Consolidation	$0.333\pm 0.000$
Unordered	$0.333\pm 0.000$
Local	$0.333\pm 0.000$
Statistics-Only	$1.000\pm 0.000$
Proposed+DeepSeek	$1.000\pm 0.000$
Oracle	$1.000\pm 0.000$

TABLE VII: Future-thinking activation after cross-experience consolidation at eight repetitions per hidden rule.

Fig. 5: Cross-experience relation discovery as the number of independent supporting episodes increases. Markers with lower opacity show individual random-seed results, while dashed lines show the mean performance. The large variation at four repetitions indicates an intermediate stage in which true relations have begun to emerge but are not yet consolidated consistently across runs.

Hong Su received the MS and PhD degrees, in 2006 and 2022, respectively, from Sichuan University, Chengdu, China. He is currently a researcher of Chengdu University of Information Technology Chengdu, China. His research interests include blockchain, cross-chain and smart contract.

Table 13

Explore similar work

PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning
Jul 22, 2026Alexis Fox, Junlin Wang, Paul Rosu +1Persistent Memory for Language Models Long-Horizon Agent Tasks
Duke University
Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoning
Feb 13, 2026Kehao Zhang, Shangtong Gui, Sheng Yang +2Persistent Memory for Language Models Long-Context Inference
1Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · University of Chinese Academy of Sciences, Beijing, China · 4Li Auto Inc. +1
Beyond Memory: Harnessing Long-Horizon Agents with Explicit Belief States
Oct 1, 2026Yu Luo, Jiamin Jiang, Yimin Zuo +9Belief Updating in LLMs Long-Horizon LLM Agents
Nankai University · Alibaba Group · Tsinghua University

Jul 22, 2026, cs.AIJ/K move · Enter open · S save

PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning

Alexis Fox, Junlin Wang, Paul Rosu, Bhuwan Dhingra

Duke University

Persistent Memory for Language Models Long-Horizon Agent Tasks LLM Context Management Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Long-horizon tasks require sustained perception, reasoning, and exploration, and are a persistent challenge for large language model (LLM) agents. This gap is reflected in their limited performance on continual learning benchmarks such as ARC-AGI-3, especially when models are evaluated out of the box. Various agent harnesses have been proposed to close this gap, and each commits to a strategy for handling long sequences of observations, i.e., what information to save from the environment and how to load it into model context, a choice we argue is particularly consequential. Existing methods for context management face a significant tradeoff, as preserving more information makes retrieving relevant details less tractable. We propose PRO-LONG, a minimal context management framework built around programmatic memory for LLM agents in long-horizon, exploratory settings. PRO-LONG addresses the tradeoff by keeping a complete, structured interaction log and capitalizing on recent progress in coding agents to search this history efficiently. On the full ARC-AGI-3 public game set, PRO-LONG improves over a base coding agent by an average of 18.0 percentage points across frontier models, and matches or exceeds state-of-the-art specialized harnesses (up to 76.1% pass@1) while using 4.2-5.8x fewer tokens. With Fable 5, PRO-LONG achieves 97.4% best@2 at a total cost of $1,750. Relevant code and logs are available at https://github.com/alexisfox7/PRO-LONG.

Abstract

Explore similar work

PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning

Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoning

Beyond Memory: Harnessing Long-Horizon Agents with Explicit Belief States