Nous: Learning and Certifying Memory Decisions Before Source Calibration

Pranav Singh

cs.LGSep 8, 2026

Nous: Learning and Certifying Memory Decisions Before Source Calibration

Authors: Pranav Singh

Organizations: Department of Mathematics, Indian Institute of Technology Ropar

Abstract

Agent memory systems update state decisions from reports whose reliability may be unknown. Existing analyses of source estimation do not determine when a policy can be learned or its improvement certified without identifying the reporting channel. We study these three tasks using the same observed records. For a specified hidden Markov family with continuous source uncertainty, decision learning and powered certification have quadratic sample complexity, whereas fixed-precision source estimation has quartic complexity. We characterize a sharp identified interval for policy gain under an unknown shared-background channel and derive finite-sample certificates under bounded history dependence and conditional copying. Independently trained witness regions support general history spaces, and disagreement-conditioned auditing improves power for sparse revisions. For dependent histories, prediction-count-preserving batches cancel the unknown reporting background and admit conditional certificates. A MultiWOZ 2.4 evaluation uses text-processing policies on 1,000 human-written test dialogues with simulated audits. Balanced batches retain 2.26 percentage points of the full candidate's 7.34 percentage-point mean gain and obtain more positive certificates under weak audits. These results establish task-specific information requirements and provide an auditable policy-revision framework for Nous.

Figures & tables

Figure 1: Population and fixed-batch revision certification in Nous. IID denotes independent and identically distributed records.

Unknown	Observable signal in the same record $O$	Task complexity
Orientation $s$	$\mathbb{E}[\chi(X_{1})Y]=3s\ell/32$	Learn: $\Theta(\ell^{-2})$
Policy improvement	$\Delta_{Y}=\ell/64$	Certify: $\Theta(\ell^{-2})$
Source scale $c$	$\mathbb{E}[X_{0,1}Y]=\ell^{2}/(16c^{2})$	Estimate: $\Theta(\ell^{-4})$

Table 1: Observable signals and sample complexities for the shared-observation family.

Figure 2: Schematic identified intervals for clean policy gain.

Figure 3: A prediction-count-preserving revision cycle.

Study	Training	Development	Calibration	Test
Mutable state	4,000	1,000	5,000	5,000
MiniGrid	–	–	5,000	1,000
Learned witnesses	4,000	4,000	8,000	5,000
MultiWOZ	8,438	1,000	–	1,000

Table 2: Data splits per run. Counts denote histories for mutable state and learned witnesses, episodes for MiniGrid, and dialogues for MultiWOZ. The learned-witness development column denotes region-selection data.

Method	Memory S7	Memory S13	S17 random
Arrival LWW	61.67 $\pm$ 1.63	60.97 $\pm$ 0.81	61.17 $\pm$ 1.50
Constant zero	50.83 $\pm$ 0.85	49.93 $\pm$ 1.27	49.70 $\pm$ 2.51
Native Nous / majority	61.23 $\pm$ 1.00	60.70 $\pm$ 0.26	61.03 $\pm$ 1.10
Root-capped Nous	75.50 $\pm$ 1.32	74.17 $\pm$ 0.72	73.63 $\pm$ 1.42
Single source 0	84.40 $\pm$ 0.75	85.23 $\pm$ 1.12	84.83 $\pm$ 0.78
Old gate vs. LWW	61.67 $\pm$ 1.63	60.97 $\pm$ 0.81	61.17 $\pm$ 1.50

Table 3: MiniGrid task success (%). Values are means with sample standard deviations across three runs. LWW denotes last-write-wins.

Figure 4: Compatible-channel acceptance frequency (%) by report signal and background bias. Stars denote nonpositive identified lower endpoints.

TV budget	Learned	Fixed coordinate	Whole history	Test gain (pp)
0.00	4/4	4/4	0/4	28.82
0.03	4/4	0/4	0/4	29.39
0.10	0/4	0/4	0/4	27.84

Table 4: Learned-witness acceptance counts and mean clean test gain (percentage points). TV denotes total variation and pp denotes percentage points.

Slot	Nous	History	Full gain	Balanced gain	Revisions kept
Hotel area	84.56	89.64	5.07	2.24	294/737
Hotel price	83.19	89.91	6.71	1.87	292/776
Hotel parking	88.28	95.40	7.12	1.65	227/725
Hotel internet	87.53	95.01	7.47	1.30	164/707
Restaurant area	81.77	90.75	8.98	3.21	460/1108
Restaurant price	82.55	92.45	9.89	3.38	426/1089

Table 5: MultiWOZ test accuracy (%), batch gain (percentage points), and retained revision counts. History denotes the full-history classifier.

Figure 5: Acceptance frequency (%, left) and mean deployed batch gain (percentage points, right) under no-copy audits.

Study	Method or comparison	Beneficial accepts	Harmful accepts
Mutable state	Unrestricted certificate	9/9	0/9
Mutable state	Report agreement	–	9/9
MiniGrid	Compatible vs. constant	9/9	–
MiniGrid	Compatible vs. LWW	4/9	–
Sparse revisions	Global certificate	1,558/3,600	0/7,200
Sparse revisions	Conditioned certificate	2,720/3,600	0/7,200

Table 6: Revision acceptances. LWW denotes last-write-wins. The sparse-revision harmful column includes null proposals. Dashes denote unreported or inapplicable comparisons.

Control outcome	Reported count or bound
Initial beneficial acceptances	0/1,800
Initial null/harmful acceptances	0/3,600
Repeated-release false acceptances	0/200
Repeated-release upper bound (%)	1.83
Phase cells with complete acceptance	23
Phase cells with partial acceptance	2

Table 7: Categorical controls, repeated releases and phase-study outcomes.

Accepted comparison	Minimum tolerance (% TV)	Maximum tolerance (% TV)
Constant incumbent	3.03	5.07
LWW incumbent	0.18	1.37

Table 8: Reported misspecification-tolerance ranges among accepted MiniGrid comparisons. TV denotes total variation and LWW denotes last-write-wins.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Full-record divergences and future-label risk in the broader model class. KL denotes Kullback–Leibler divergence and ERM denotes empirical risk minimization.

Method	Independent	Copy	Delayed copy
Arrival LWW	74.82 $\pm$ 0.39	68.41 $\pm$ 0.91	58.34 $\pm$ 0.50
Event-time latest	74.82 $\pm$ 0.39	68.41 $\pm$ 0.91	67.91 $\pm$ 0.43
Original Nous updater	77.86 $\pm$ 0.14	67.72 $\pm$ 0.94	55.81 $\pm$ 0.86
Root-deduplicated Nous	77.86 $\pm$ 0.14	74.51 $\pm$ 0.71	73.78 $\pm$ 0.81
Current anchor	85.31 $\pm$ 0.26	80.10 $\pm$ 0.92	80.03 $\pm$ 0.81
Current majority	89.52 $\pm$ 0.23	67.92 $\pm$ 0.79	67.56 $\pm$ 0.45

Appendix

Table S1: Mutable-state accuracy (%). Values are means with sample standard deviations. LWW denotes last-write-wins and HMM denotes hidden Markov model.

Scenario	Run	Gain (pp)	95% CI (pp)	Harmful / beneficial
independent	1	19.76	[18.52, 21.00]	102 / 1090
independent	2	18.86	[17.59, 20.13]	144 / 1087
independent	3	18.82	[17.56, 20.08]	136 / 1077
copy	1	15.66	[14.29, 17.03]	277 / 1060
copy	2	17.52	[16.15, 18.89]	249 / 1125
copy	3	15.38	[13.94, 16.82]	350 / 1119

Appendix

Table S2: Selected policy gain and individual revision outcomes. LWW denotes last-write-wins, pp denotes percentage points, and CI denotes confidence interval.

Environment/run	Incumbent	Accepted	Gain (pp)	Paired 95% CI
S13/1	LWW	0	0.00	[0.00, 0.00]
S13/1	Constant	1	23.70	[20.46, 26.94]
S13/2	LWW	0	0.00	[0.00, 0.00]
S13/2	Constant	1	24.10	[20.90, 27.30]
S13/3	LWW	1	13.10	[10.50, 15.70]
S13/3	Constant	1	24.90	[21.60, 28.20]

Appendix

Table S3: MiniGrid deployed gains and paired confidence intervals. LWW denotes last-write-wins, pp denotes percentage points, and CI denotes confidence interval.

Figure S2: MiniGrid lower bounds under increasing assumed channel deviation. TV denotes total variation and LWW denotes last-write-wins.

$n$	$d$	Global bound	Conditioned bound
2000	1	300	300
2000	.25	58	300
2000	.05	0	21
2000	.01	0	0
8000	1	300	300
8000	.25	300	300

Appendix

Table S4: Beneficial acceptance counts by calibration size

n

and disagreement probability

d

Slot	Nous	Last	Frequency	Recency .5	Recency .8	History
Hotel area	84.56	83.78	83.41	84.01	83.41	89.64
Hotel price	83.19	82.76	81.17	82.39	81.31	89.91
Hotel parking	88.28	88.01	86.46	88.23	87.38	95.40
Hotel internet	87.53	87.57	86.07	87.75	86.91	95.01
Restaurant area	81.77	79.96	79.63	80.53	79.94	90.75
Restaurant price	82.55	80.41	79.58	81.48	80.39	92.45

Appendix

Table S5: All-turn slot accuracy (%) by memory policy.

Slot	Specified values		State changes
	Nous	History	Nous	History
Hotel area	22.83	52.52	27.75	40.53
Hotel price	21.68	52.90	29.77	48.09
Hotel parking	28.80	73.07	61.11	65.00
Hotel internet	23.75	70.55	54.26	71.28
Restaurant area	39.61	72.17	53.73	60.90

Appendix

Table S6: Slot accuracy (%) for specified values and state-change turns.

Slot	Nous	Last	Frequency	Recency .5	Recency .8
Hotel area	99.19	100.00	93.92	100.00	97.42
Hotel price	98.93	100.00	93.42	100.00	97.44
Hotel parking	99.24	100.00	94.74	100.00	97.73
Hotel internet	99.19	100.00	94.79	100.00	97.79
Restaurant area	99.27	100.00	93.45	100.00	97.25
Restaurant price	99.09	100.00	93.91	100.00	97.74

Appendix

Table S7: All-turn slot accuracy (%) with oracle extraction.

Policy	Brier	Log loss
native nous	0.2574	1.9914
last report	0.2496	0.5308
frequency	0.2273	0.8931
recency 0.5	0.2617	0.5554
recency 0.8	0.3807	0.7987
history classifier	0.1137	0.2266

Appendix

Table S8: Mean Brier score and log loss across categorical slots.

$\rho$	$\epsilon$	Balanced	Count-only	Oracle	Naive harmful
0.3	0.00	241	98	691	582
0.3	0.03	102	31	642	677
0.3	0.10	5	0	328	700
0.6	0.00	688	629	700	0
0.6	0.03	656	523	700	0
0.6	0.10	404	192	699	0

Appendix

Table S9: Forward acceptance counts and naive harmful reverse acceptances. Rows pool seven slots and 100 draws per slot.

Setting	Value
Mutable-state runs	9
Mutable-state snapshots and sources per snapshot	4 and 5
Mutable-state copy probability	$.9$
Native Nous declared reliability	$.75$
HMM expectation-maximization iterations and restarts	25 and 3
Mutable-state audit $(q_{0},q_{1})$	$(.45,.95)$

Appendix

Table S10: Evaluation settings. HMM denotes hidden Markov model.

Replay quantity	Count or value
Mutable-state held-out queries	45,000
Selected HMM and suffix-two runs	7 and 2
Mutable-state split seeds	36
Mutable-state runtime (seconds)	42.55
MiniGrid test episodes	9,000
MiniGrid reconstructed certificates	36

Appendix

Table S11: Evaluation and replay quantities. Runtime is measured on the local host.

Text-model or audit setting	Value
Maximum features per view	35,000
Maximum classifier iterations	50
Classifier regularization	$10^{-5}$
Classifier convergence tolerance	$10^{-4}$
Recency decay factors	$.5,.8$
Candidate policies	6

Appendix

Table S12: Text-model and audit settings.

Proposal	Retained revisions	Mean clean gain (pp)
Full candidate	5,987	7.34
Scored balanced	2,236	2.26
Count-only balanced	2,243	1.77

Appendix

Table S13: Retained MultiWOZ revisions and mean clean gains. The abbreviation pp denotes percentage points.

Reference signal	Scored balanced gain (pp)	Full corrected gain (pp)
$.6$	2.23	0
$.9$	2.26	5.25

Appendix

Table S14: Mean deployed gain under no-copy audits. The abbreviation pp denotes percentage points.

Dataset	Incumbent accuracy (%)	Candidate accuracy (%)	Outcome
EEG	70.69	37.07	Harmful acceptance
Occupancy	55.36	99.49	Beneficial rejection

Appendix

Table S15: Previously exposed time-series diagnostics. EEG denotes electroencephalography.

Calculation quantity	Value
Specified laws	27
Null or harmful laws	18
Largest null or harmful acceptance probability	$1.16\times 10^{-21}$

Appendix

Table S16: Conditional-binomial acceptance calculation.

Explore similar work

The Delegation Blind Spot: Auditing Product Decisions from Agent Choices
Sep 22, 2026Shivam GuptaAI Agent Auditing Partial Identification
Independent Research
Self-Certification of Representation Adequacy: Sequential Certification at Minimum Task Loss
Aug 3, 2026Zijie HuangState Representation Learning Statistical Learning Theory
Independent Researcher
Auditing Near-Optimal Policies Can Be Exponentially Hard: Conditional Query Lower Bounds via Occupancy Rashomon Capacity
May 29, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaInformation-Theoretic Lower Bounds Black-Box Model Auditing
Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University

Sep 22, 2026, cs.AIJ/K move · Enter open · S save

The Delegation Blind Spot: Auditing Product Decisions from Agent Choices

Shivam Gupta

Independent Research

AI Agent Auditing Partial Identification Decision-Making under Uncertainty Black-Box Model Auditing

Open paper AskPDF arXiv

Successful agent execution need not identify which future product improvement its user would value. We present a decision-specific audit that maps a declared observation channel and product-value contrast to compatible intervals and witness populations. Its foundations are established identification and decision theory; the contribution is an executable measurement workflow and a controlled study of its limits. A frozen experiment makes 4,800 requests to two pinned model snapshots on shared synthetic tasks. All 36 conservative primary intervals remain unresolved despite different execution accuracy. An exploratory 2,400-call follow-up records supplied preferences and resolves three of nine comparisons per model. A deterministic extractor resolves seven of nine without model calls or calibration observations, exposing unnecessary uncertainty introduced by model-generated reports. A further 14,400 controlled multinomial simulations distinguish structural ambiguity from weak identification and finite calibration precision. We propose a source-labeled decision receipt and provide an offline viewer for inspecting the audit. These results motivate preserving decision-relevant structured input and diagnosing why a decision is unresolved before collecting more telemetry. The study contains no human participants or real customer outcomes. Full proofs, raw model provenance, controlled experiments, and reproducible analyses accompany the report.

Abstract

Explore similar work

The Delegation Blind Spot: Auditing Product Decisions from Agent Choices

Self-Certification of Representation Adequacy: Sequential Certification at Minimum Task Loss

Auditing Near-Optimal Policies Can Be Exponentially Hard: Conditional Query Lower Bounds via Occupancy Rashomon Capacity