TrackMem: Tracking Evolving Beliefs in Long-Horizon Financial Dialogue

Austin M. Y. Cheung; Harsh Vardhan Gupta; Go Nam Lui; Lok Kan Chan; Ginny Y. Wong; Simon See

cs.CLNov 1, 2026

TrackMem: Tracking Evolving Beliefs in Long-Horizon Financial Dialogue

Authors: Austin M. Y. Cheung, Harsh Vardhan Gupta, Go Nam Lui, Lok Kan Chan, Ginny Y. Wong, Simon See

Organizations: The Hong Kong University of Science and Technology · FinCatch Ltd · City University of Hong Kong · NVIDIA AI Technology Center (NVAITC), NVIDIA

Abstract

Memory systems for LLM agents are evaluated mostly on social dialogue, where facts accumulate and rarely change, so remembering reduces largely to finding the turn in which something was said. Financial dialogue breaks this assumption: figures are revised, stances shift without being restated, rules stated once govern later questions, and a single company is discussed under several analytical lenses. Memory systems that retrieve the passages most similar to a question, or the most recent ones, therefore return superseded values, miss rules that share no wording with the question, and cannot recover a stance spread across sessions. We present TrackMem, a per-speaker graph memory that tracks the history of figures, separates perspectives by entity and lens, and preserves each speaker's actions in chronological order. To evaluate it, we introduce FinLoCoMo, a benchmark of 15 long-horizon investment conversations with 765 questions in two splits: one following the LoCoMo categories and one targeting four failure modes. With two readers, TrackMem outperforms existing memory systems on both splits. On the failure-mode split it reaches 73.9% macro accuracy against 69.4% for the strongest baseline, with the largest gains on questions about revised figures (86.5% against 73.4%).

Figures & tables

Figure 1: Evidence recall against context length.

System	Update	Revision	Stance	Constraint	Aspect
Turn RAG	append	○	○	○	○
MemGPT	self-edit	◐	○	◐	○
Mem0	overwrite	◐	○	○	○
A-Mem	note links	○	○	○	○
Zep	invalidation	◐	○	○	○
O-Mem	attributes	○	○	◐	◐

Table 1: Failure modes each system addresses by design. ● addressed; ◐ partially; ○ not addressed.

Figure 2: Illustrative example from a long-horizon financial conversation. TrackMem stores each kind of information in its own structure and retrieves it by entity and kind. Recall is evaluated based on FinLoCoMo-Modes.

Standard split (464)		Modes split (301)
Single-hop	168	Revision	64
Multi-hop	76	Stance	50
Temporal	74	Constraint	148
Open-domain	73	Aspect	39
Adversarial	73

Table 2: FinLoCoMo questions (765 in total). Conversations average 27.3 sessions of 9.0 turns and 7.9k tokens.

Figure 3: Design of TrackMem

FinLoCoMo-Modes
	Retrieval		Gemini 3.1 Flash Lite						Nemotron 3.5 Lightning
Method	Recall	Tokens	Rev.	Stance	Con.	Aspect	Macro	Micro	Rev.	Stance	Con.	Aspect	Macro	Micro
			(64)	(50)	(148)	(39)		(301)	(64)	(50)	(148)	(39)		(301)
Mem0	0.42	311	54.0	12.8	66.3	27.0	40.0	50.0	48.4	15.0	47.3	32.1	35.7	40.2
A-Mem	0.70	3,687	69.4	56.1	79.3	59.2	66.0	70.7	76.6	61.0	80.7	65.4	70.9	74.6
MemGPT	0.55	2,656	73.4	55.1	76.5	72.4	69.4	71.8	80.5	66.0	76.4	83.3	76.5	76.4

Table 3: Main results (%). Rev. = Revision , Con. = Constraint . Recall : share of annotated evidence turns in the context. Micro is over successfully judged questions (range from 0 – 18 unjudged per cell).

Upstream ablation
	Modes	Standard
Full memory	74.4	70.3
$-C$	70.6 $\downarrow$ 3.8	68.8 $\downarrow$ 1.5
$-A$	73.2 $\downarrow$ 1.2	69.3 $\downarrow$ 1.0
$-P$	74.0 $\downarrow$ 0.4	68.9 $\downarrow$ 1.4
Context ablation

Table 4: Micro accuracy after removing one component. Each block is a separate build and judge run, so its first row differs from Table 3 .

In production since	July 2026
Ingested turns	13,781
Nodes per user	604
Estimated model write cost per batch	$\approx\$ 0.005$
Ingestion delay (median)	19.0 min
Estimated query latency	3–5 s
Batches entering dead-letter queue	0.13%

Table 5: TrackMem deployment (Oct 2026).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	Mean	Min	Max
Sessions	27.3	24	28
Turns	247	196	282
Turns per session	9.0	2	22
Days spanned	411	191	612
Turn text (tokens)	7.9k	5.6k	9.5k

Appendix

Table 6: Per-conversation statistics of FinLoCoMo.

Category	$n$	Evidence	Sessions	Span
Single-hop	168	1.11	1.05	0.08
Multi-hop	76	2.32	1.54	0.83
Temporal	74	2.01	1.95	1.78
Open-domain	73	5.07	3.25	3.29
Revision	64	4.25	3.55	9.03
Stance	50	3.90	2.60	5.36

Appendix

Table 7: Mean evidence turns, sessions cited and session span per question, by category.

Category	Generated	Kept	Blind	Other
Revision	78	38	23	17
Stance	72	33	31	8
Constraint	71	26	41	4
Aspect	74	9	22	43
All	295	106	117	72

Appendix

Table 8: Candidates removed by each filter in an earlier build of the Modes split. Blind : failed the blind check. Other : failed the structural, chance-floor or duplicate check.

Type	Question	Evidence	Retrieval trap	Gold
Revision	What discount rate is Sarah currently using for her SPG DCF?	D16:3 “Using 8.0% for SPG now, up from 7.75%.” D17:2 “10Y near 4.2% pushes my discount rate to 8.25%.”	D17:6 , the latest mention: “at 4.3% I’d bump SPG’s discount rate up another 25bps” (hypothetical)	8.25%
Stance	What data point caused Robert to first raise his XOM price target above $126?	D21:3 “ $126 target still works if Brent holds here.” D22:3 “$ 94 Brent has me second-guessing my mid-cycle call … XOM’s $126 target needs a$ 5–10 bump.” D23:5 “updated my PT to $131 from$ 126.”	D23:5 states the raise but not its cause; D25:3 reports a later raise to $136	Brent reaching $94, prompting a late-cycle rethink
Constraint	Given Kenji’s trailing-stop discipline, at what price would his stop trigger on the Sony position he reloaded at ¥13,420?	D21:5 “30bps trailing, same as before … not giving back more than 0.3% on any single name.” D26:5 “picked up 1% Sony last Wednesday at ¥13,420.”	D26:5 matches the question but carries no rule; the rule is five sessions earlier	$\approx$ ¥13,380
Aspect	How does Eleanor reconcile liking SPG’s valuation with her reluctance to size it up, and what risk lens dominates her hesitation?	D25:4 “SPG feels like the steadier play—100 shares there is sleep-at-night money.” D26:4 “Same CEO across two 90%+ payout names is a concentration risk red flag.” D26:10 “my 100 SPG shares at 6.65% yield still feel like the right anchor.”	D22:5 cites a different risk: “100 SPG shares funding that 6.65% yield with debt has me nervous”	Likes the 6.65% yield but holds at 100 shares; same-CEO concentration risk

Appendix

Table 9: Example questions from the Modes split. Evidence lists the turns (session:turn) the gold depends on; Retrieval trap is a turn that a similarity- or recency-based retriever is likely to return instead.

L1 domain	L2 subtopics
Fundamental analysis	Income statement, Balance sheet, Cash flow, Valuation, Management quality
Market dynamics	Price action, Institutional flow, Sentiment, Liquidity
Corporate actions	Earnings events, M&A activity, Capital structure, Management changes
Quantitative & risk	Factor exposure, Portfolio risk, Earnings quality
Macro & sector	Macro environment, Sector dynamics, Geopolitical risk, Commodities & FX

Appendix

Table 10: L1 and L2 levels of the topic taxonomy.

Field	Values
Anchor kind, commitment	position cap, position size, add level, trim level, price target, valuation multiple, yield, estimate
Anchor kind, observation	observed level, third-party estimate, executed fill, P&L
Unit	currency, percent (basis points are normalised to percent), multiple, shares, quantity
Comparator	at, below, above
Source	user, endorsed (the other party’s figure, adopted by the user), counterparty, third party
Action kind	initiated, added, trimmed, exited, upgraded, downgraded, target raised, target cut, stop tightened, stop loosened, hedged, deferred, held

Appendix

Table 11: Closed vocabularies of the extraction schema.

Grounding	$\lambda_{e}$	$\lambda_{z}$	$\lambda_{t}$
Entity-anchored	0.10	0.02	0.02
Topic-anchored	0.03	0.08	0.05
Semantic-only	0.02	0.02	0.01
None returned	0.05	0.03	0.02

Appendix

Table 12: Weights of the entity, topic and tag terms of the view score, by the question’s grounding. The values are set by hand.

Figure 4: Write and read paths of the deployed TrackMem .

	Turn RAG	TrackMem	$\Delta$
revision	70.2	86.5	$\uparrow$ 16.3
stance	65.3	61.5	$\downarrow$ 3.8
Constraint	74.8	79.1	$\uparrow$ 4.3
aspect	63.2	68.4	$\uparrow$ 5.2
Macro	68.4	73.9	$\uparrow$ 5.5
Recall	0.82	0.80

Appendix

Table 13: Equal retrieval budget on Modes: similarity retrieval over turns (no graph) and TrackMem , each showing about 27 turns.

	Gemini			Nemotron
Method	Prec.	Rec.	F1	Prec.	Rec.	F1
Mem0	26.0	93.2	40.6	23.8	98.6	38.4
A-Mem	69.1	89.0	77.8	59.2	83.6	69.3
MemGPT	81.3	83.6	82.4	80.0	82.2	81.1
Graphiti (Zep)	19.0	71.2	30.0	15.5	82.2	26.1
O-Mem	22.5	52.1	31.4	21.9	57.5	31.7

Appendix

Table 14: Precision and recall of a decline. A true positive is a decline on one of the 73 unanswerable questions, counted over all 765 questions.

Call	Model	Max	Output fields
Extraction	Gemini	65,536	entities, relations, anchors, claims, attributes
Synthesis	Gemini	8,192	text and sentiment per (entity, topic) row
Resolver $\psi$	Gemini	1,024	stored quantity revised, or none
Router	Gemini	1,024	entities, topics, type, date, grounding
Reader	Gemini	1,024	value, answer, declined, support
Reader	Nemotron	8,192	as above

Appendix

Table 15: Model calls, all at temperature 0. Max : maximum output tokens. Gemini is gemini-3.1-flash-lite and Nemotron is nemotron-3.5-lightning . The first four calls build or query the memory of TrackMem (Appendix E.1 – E.4 ); the reader and judge are shared by all methods. The resolver runs only when a new commitment has stored candidates.

Long-Term Conversational Memory

Long-term conversational memory helps AI assistants retain, update, and use information from past conversations across sessions.

See all 276 papers

FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents
Aug 4, 2026Ben Wang, Kang Zhou, Lifan Guo +2Financial Services LLM Agent Memory
Qwen DianJin Team, Alibaba Cloud Computing
LeanMem: Simple and Efficient Long-Term Memory for LLM Agents
Aug 4, 2026Yuxin Liao, Le Wu, Min Hou +3LLM Agent Memory Persistent Memory for Language Models
Hefei University of Technology
HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents
Apr 20, 2026Shuqi Cao, Jingyi He, Fei TanPersistent Memory for Language Models Hierarchical Memory
East China Normal University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China

Aug 4, 2026J/K move · Enter open · S save

FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents

Ben Wang, Kang Zhou, Lifan Guo, Feng Chen+1

Qwen DianJin Team, Alibaba Cloud Computing

Financial Services LLM Agent Memory Memory Agent Benchmarks Finance

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly used as personalized assistants in high-stakes domains such as financial advising, yet it remains unclear whether they can maintain and update an individualized user model over long horizons. Existing personalized-memory benchmarks primarily test factual retention or rely on weakly constrained model-generated trajectories, leaving event-driven preference adaptation underexplored. We introduce FinPerMA, an event-grounded benchmark that evaluates personalized memory against frozen longitudinal investor trajectories. Its generation pipeline combines deterministic, theory-informed impact rules, controlled LLM narration, and automated quality screening; a Post-Shock checkpoint isolates whether an agent has integrated a material event into its persistent user model. On 2,994 questions from 276 personas, seven frontier LLMs and up to seven memory configurations remain far from saturated: no full-context configuration exceeds approximately 0.47 overall accuracy or approximately 39% on multiple-choice questions. Attribution analysis shows that summary-based memory often preserves factual details while losing the preference signals needed for personalization; simple retrieval can therefore outperform purpose-built memory systems, with the gap widening after shocks.

Abstract

Long-Term Conversational Memory

Explore similar work

FinPerMA: A Theory-Informed, Event-Grounded Personalized-Memory Benchmark for LLM Agents

LeanMem: Simple and Efficient Long-Term Memory for LLM Agents

HiGMem: A Hierarchical and LLM-Guided Memory System for Long-Term Conversational Agents