Beyond Update Counts: A Control Map for Evaluating Test-Time Model Updates

Zehao Zhao; Shangjian Yin; Rui Liu; Chongyang Bai; Mark Ibrahim; Kavosh Asadi; Yuchen Lu; Hang Cui

cs.LGOct 26, 2026

Beyond Update Counts: A Control Map for Evaluating Test-Time Model Updates

Authors: Zehao Zhao, Shangjian Yin, Rui Liu, Chongyang Bai, Mark Ibrahim, Kavosh Asadi, Yuchen Lu, Hang Cui,

Organizations: Meta AI

Abstract

Test-time updating lets a language model change persistent state after training---such as model weights, adapter parameters, or a memory module---before or while answering queries. Existing evaluations often compare systems using update count or frequency and performance measured immediately afterward. These summaries omit what is updated, where, when, and in what order. The same total can therefore hide an important difference: one system may answer after one update while another answers after two. We introduce the Control Map, an evaluation framework that records the committed updates preceding each scored answer and separately evaluates behavioral changes, immediate learning (acquisition), later recall (retention), preservation of unrelated behavior (locality), and performance in new settings. We prove when counts alone determine final state and show, under stated assumptions, how differences between pre-answer histories limit differences in average scores. Using ControlMap, our experiments reveal three insights. First, update scheduling matters: at a 760M-parameter checkpoint, 24 schedules with the same update count yield 19.7%--34.2% accuracy. Second, learning an edit does not guarantee that it lasts or remains contained: all six selected CounterFact LoRA settings learn the new information, while ROME retains edits at all 13 operating points but fails locality at each. Third, no editor performs best consistently across models, benchmarks, and tests. Evaluations should therefore report update histories and assess acquisition, retention, and locality separately.

Figures & tables

Figure 1: The same update count can yield different histories at an answer. Both plans perform $u_{1}$ and $u_{2}$ , but only plan B performs $u_{2}$ before scoring, so their recorded histories are $(u_{1})$ and $(u_{1},u_{2})$ under the stated tracing rule. The Control Map reports update history, measured effect, memory quality, and results in a new setting separately. The formal record also stores cost; values are schematic.

Figure 2: Where updates occur changes performance even when update count is fixed. All 24 assignments of periods

\{1,2,4,8\}

to four ordered layer bands are evaluated on the same 6,000-example single-needle exact retrieval (Single-NIAH) evaluation set. Schedule codes list periods from the earliest to the latest layer band; whiskers are pointwise paired 95% bootstrap ranges over the fixed evaluation set. The additive assignment map explains 95.3% of between-schedule variation descriptively (leave-one-schedule-out

R^{2}=0.863

, root-mean-square error

=1.59

percentage points). Best–worst labels and the decomposition are retrospective, so this is a local response surface rather than a universal schedule ranking.

Choice	Comparison and controls	Observed result	Supported takeaway
Placement	24 band–period assignments; one MAC-760M checkpoint; 6,000 paired Single-NIAH rows; equal prompt-only writes	Accuracy $19.733\%$ – $34.233\%$ ; the band–period model descriptively explains $95.322\%$ of between-schedule variation	Placement matters in this task and checkpoint;
Timing	Same selected events and terminal actions; writes before versus after the focal query	Immediate-minus-deferred acquisition: $+0.1196,+0.0598,+0.4320,$ and $+0.1688$ nats across four GPT-2 sizes ( Radford et al., 2019 ) ; Pythia-410M ( Biderman et al., 2023 ) : $+0.1592$ nats, range $[-0.4484,+0.7667]$	The estimated effect is positive in all four GPT-2 assays; the Pythia range crosses zero
Dose	Eight objectives, eight actions, 32 stochastic-gradient-descent steps, and terminal adapter–optimizer construction fixed	Across 18 deliberately varied LoRA settings, the best tested doses $1/2/4/8$ occur in $9/7/2/0$ settings; the best-dose prediction specified before outcomes is confirmed in 7/18 settings	The best tested dose varies across settings

Table 1: Controlled comparisons of placement, timing, and dose. Each row changes one choice while preserving the listed controls; conclusions remain specific to the tested setting.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Claim / audit question	Study	Timing	Unit and scale	Location
Equal counts need not identify equal executed treatments	MAC-760M trace atlas; 24 depth–cadence schedules on eight panels	Outcome-blind exact enumeration	27,072 benchmark rows; 24 policies per row	Section B ; Table 6
Layer placement can change accuracy at fixed update count	MAC-760M placement surface on Single-NIAH	One fixed checkpoint; additive summary descriptive	6,000 paired rows; all 24 schedules	Figure 2 ; Section B.4
Order can masquerade as dose–locality structure	GPT-2XL LoRA complete-order correction	Post-hoc deterministic correction	1,600 order-matched pairs	Section B
Query-relative phase can change acquisition	GPT-2 family and Pythia-410M phase studies	Per-model frozen gates; cross-model synthesis descriptive	Four LoRA initializations per model	Section B
The best tested dose varies across settings	LoRA/SGD four-dose grids	Purposive, partly adaptive setting collection; within-setting calls frozen	18 settings; four adapter initializations per setting	Section B.5
Acquisition need not imply feasible memory	LoRA and ROME grids; fixed MEMIT, ROME, and AlphaEdit controls	Cohort-specific frozen or declared timing	Family/block resampling within each assay	Table 7 ; Sections C.2 – C.10

Appendix

Table 2: Studies and supporting evidence. Each row identifies the study, analysis timing, inference or enumeration unit, and supporting section. Assays answer separate questions and are reported separately. “Frozen” means fixed and recorded before the named outcome was accessed; these protocols were not publicly preregistered.

Declared aggregate deviation $\bar{\epsilon}$	Binary-effect cap (points)
$0$	$0.0512$
$0.001$	$0.1512$
$0.010$	$1.0512$
$0.050$	$5.0512$

Appendix

Table 3: SocialIQA certificate sensitivity. The right column is

100\min\{1,1/1954+\bar{\epsilon}\}

percentage points.

Study	Recorded boundary	Conformance evidence	Ancestry-coverage claim
MAC trace atlas	Declared deterministic scheduler graph	Complete 27,072-row enumeration; 184 outcome-blind checks	Exact relative to the declared scheduler adapter; no whole-program claim
Constructed neural LoRA test	Recorded live state transitions and nonzero score-gradient paths	16/16 chains validate; $s_{Z}=1/2$ ; 16 checks reject and restoration passes	Conformance for recorded paths in one randomly initialized small Llama; unrecorded dependencies remain unverified
GPT-2XL LoRA phase study	Declared PEFT-LoRA/SGD event and optimizer-step trace	Separate implementation matches 3,072/3,072 fields; 15/15 tests reject five omission/alias/substitution fault classes	Structural reconstruction only; no live-state observation or completeness theorem
WikiRecent	Score-ancestral editor outer calls	512/512 policy rows match precomputed certificates	EasyEdit-internal false-positive/negative rates unavailable
GPT-2XL endpoints	Reconstructed editor outer-call traces	448/448 compiled cells agree; 1,792 family–order units and 3,584 policy traces match	Editor internals were not observed
Llama-3.2-3B endpoints	Declared dose-four/terminal atomic outer-call traces	Post-outcome projection: 3,584/3,584 rows conform; all 1,792 paired traces separate ( $s_{Z}=1$ )	EasyEdit internals and cross-bundle correspondence unavailable; diagnostic component only

Appendix

Table 4: Conformance and coverage. Counts establish conformance only at the level named in the second column.

Context	Policy	fail	inconclusive	pass	At $d=5$
New 160-family CounterFact panel	MEMIT-high	$d<1.125000$	$1.125000\leq d\leq 3.203125$	$d>3.203125$	pass
New 160-family CounterFact panel	ROME-low	$d<3.078125$	$3.078125\leq d\leq 6.421875$	$d>6.421875$	inconclusive
WikiRecent shared locality	MEMIT-high	$d<33.984375$	$33.984375\leq d\leq 52.734375$	$d>52.734375$	fail
WikiRecent shared locality	ROME-low	$d<32.2265625$	$32.2265625\leq d\leq 50.781250$	$d>50.781250$	fail

Appendix

Table 5: Locality verdict as a function of damage tolerance

d

. Endpoint records are not pooled. Bounds are percentage points.

Figure 3: Resolution depends on the pre-answer write horizon. Exact enumeration yields 1, 4, 12, or 24 distinct histories as the number of complete writes available before an answer increases. The right panel shows the resulting mixture for each benchmark, with each bar normalized by that benchmark’s row count. The calculation covers all 27,072 rows and is deterministic, so no sampling intervals apply.

Panel	Rows	Pair alias	Full rows	Rank/9	$\kappa(I_{U})$
SocialIQA	1,954	14.31%	0.05% (1 row)	9	1954.000
BoolQ	3,270	0.15%	96.61%	9	1.035
LAMBADA	5,153	0.01%	99.77%	9	1.002
PIQA	1,838	66.60%	0.00%	6	—
WinoGrande	1,267	43.58%	0.00%	6	—
HellaSwag	10,042	5.84%	40.60%	9	2.445

Appendix

Table 6: Complete execution atlas. “Pair alias” is the mean fraction of 276 nominal schedule pairs with equal rowwise traces. “Full rows” is the percentage of rows that distinguish all 24 policies. Rank and condition number

\kappa

refer to the centered nine-dimensional band–period working basis and diagnose design support, not causal identification or sampling power.

Assay	Acquisition	Retention	Locality	Full gate
LoRA, six selected CounterFact settings	6/6 pass	0/6 pass	0/6 pass	0/6 pass
ROME, 13 CounterFact operating points	13/13 pass	13/13 pass	0/13 pass	0/13 pass
MEMIT-high, pilot-disjoint 160-family CounterFact point	pass	pass	pass	pass
ROME-low, same 160-family point	pass	pass	inconclusive	inconclusive
AlphaEdit, shared-eight-event CounterFact doses	4/4 pass	4/4 pass	4/4 pass	4/4 pass

Appendix

Table 7: Acquisition cannot stand in for feasibility. Rows summarize separate assays with their own counts, horizons, cohorts, and inference units. A full pass requires every behavioral condition specified for that study; cost is reported separately because no common budget was declared.

Figure 4: Dose-one writer profiles differ across evaluation dimensions. The threshold-centered display uses a shared eight-event CounterFact cohort with 16 disjoint family-pair blocks per writer (32 focal orientations and four event orders; 128 rows per writer). Outward is favorable, the dashed radius marks each axis’s threshold, and the embedded table reports values on their native scales. Dose one indexes query-time acquisition; delayed recall and locality are measured after the shared eight-event sequence. Formal acquisition, delayed-recall, and locality decisions use recorded intervals; installation and keep-among-installs use point thresholds, and those intervals are not displayed on the spokes. The axes are separate requirements, so polygon area is not an overall utility or an editor-family ranking. FT-L denotes the constrained fine-tuning baseline. Because FT-L has no installations, its displayed 0.00 at Keep $\mid$ install is a layout placeholder; the formal conditional-retention field is undefined .

Figure 5: Acquisition and locality expose different dose behavior on the shared eight-event cohort. Panel A juxtaposes query-time acquisition with terminal locality at the matched dose-four slice and shows separate marginal finite-panel ranges; filled markers pass the full declared gate. Panel B shows query-time acquisition at the four measured doses

\{1,2,4,8\}

only. The Pareto annotation and connecting lines are descriptive: they neither identify a causal dose trade-off, interpolate between doses, nor define an uncertainty-adjusted frontier.

Setting	Call	Peak	Acq. [95% range]	Ret. [95% range]	Install	Keep $\mid$ install	Locality [95% range]
Pythia current	1	1	$+8.481$ $[+7.531,+9.473]$	$-10.947$ $[-13.035,-8.853]$	$0.8125$	$0.0096$	$-37.03$ $[-49.61,-24.53]$
Pythia step100000	1	2	$+6.666$ $[+5.781,+7.572]$	$-12.029$ $[-13.529,-10.609]$	$0.3906$	$0.0000$	$-38.44$ $[-48.83,-27.42]$
GPT-2 Large	4	4	$+4.463$ $[+3.841,+5.123]$	$+4.463$ $[+3.841,+5.123]$	$0.0781$	$0.9000$	$-53.20$ $[-62.81,-43.20]$
GPT-2XL	4	4	$+2.746$ $[+2.041,+3.477]$	$+2.746$ $[+2.041,+3.477]$	$0.0547$	$0.7143$	$-41.95$ $[-51.88,-32.03]$

Appendix

Table 8: Full Stage-0 decision at each pre-outcome-frozen predicted dose. “Call” is the frozen selected dose, “Peak” is the empirical maximizing dose, and “Keep

\mid

install” is retention among installed pairs. “Acq.” and “Ret.” denote acquisition and retention. Log-probability contrasts are in nats; locality is in percentage points.

Setting	$d_{1}$	$d_{2}$	$d_{4}$
Pythia current	$+8.481$	$+3.225$	$-6.114$
Pythia step100000	$+6.666$	$+6.674$	$-7.497$
GPT-2 Large	$+0.520$	$+1.340$	$+4.463$
GPT-2XL	$+0.342$	$+1.101$	$+2.746$

Appendix

Table 9: Mean acquisition contrast across the complete Stage-0 dose grid. These are descriptive point means; the frozen decisions use the predicted-dose family-bootstrap ranges in Table 8 .

Model	$A(d_{1})$ [95% range]	$A(d_{2})$	$A(d_{4})$	Retention (nats) / $L$ (pp)
Qwen2.5-0.5B	$6.145$ $[4.737,7.556]$	$-2.212$	$-4.366$	$-7.029$ / $-30.625$
Llama-3.2-1B	$4.581$ $[3.837,5.358]$	$0.333$	$-1.671$	$-4.143$ / $-38.359$

Appendix

Table 10: Family-extension acquisition surfaces and metrics at the frozen dose-1 call.

A

is acquisition, retention is a log-probability contrast in nats, and

L

is terminal-minus-reset neighborhood success in percentage points.

Editor	Native strength	$d=1$	$d=2$	$d=4$	Frozen $A$ -response label
MEMIT	high	$7.185/1.875$	$7.196/1.875$	$7.205/1.875$	dose-insensitive
ROME	low	$6.376/5.391$	$6.513/5.469$	$6.854/6.328$	inconclusive
ROME	default	$8.118/18.281$	$8.167/20.312$	$8.201/25.547$	dose-insensitive
ROME	high	$8.611/35.391$	$8.625/38.906$	$8.582/41.797$	dose-insensitive

Appendix

Table 11: GPT-2XL published-editor schedule grids.

A

is acquisition at the dose query in nats and

D

is terminal locality damage after all four writes in percentage points (lower is safer). Each entry is

A/D

. Native strength values are not commensurate across editors.

Dose	$A$ [95% range]	$D$	Install	Retain/install	Full gate
$d=1$	$7.18483\ [6.21130,8.08745]$	1.09375	0.375	1.000	pass
$d=2$	$7.19700\ [6.22382,8.09705]$	1.09375	0.375	1.000	pass
$d=4$	$7.21356\ [6.25123,8.10839]$	1.09375	0.375	1.000	pass
$d=8$	$7.26836\ [6.30097,8.17701]$	1.09375	0.375	1.000	pass

Appendix

Table 12: MEMIT-high on the separate shared-eight-event GPT-2XL facet.

A

is query-time acquisition in nats and

D

is terminal locality damage in percentage points. Brackets are pointwise 95% block-reweighting ranges; the max–min range and dose-eight-minus-dose-four comparison below use the frozen max-deviation band.

Point	$A$ [95% range]	$D$	$L$ range	Install	Keep $\mid$ install
MEMIT-high	$7.980$ $[7.272,8.651]$	$3.750$	$[-7.969,-0.781]$	$0.508$	$1.000$
ROME-low	$7.522$ $[6.685,8.264]$	$7.148$	$[-11.523,-3.086]$	$0.457$	$0.966$

Appendix

Table 13: Fresh 64-family confirmation.

A

is post-fourth-edit minus reset target log probability;

L

is terminal-minus-reset neighborhood success, so

D=-L

is locality damage. All brackets are pointwise 95% family-reweighting ranges; editor contrasts are formed within family before reweighting.

Editor	$A$ [95% range]	$D$ ( $L$ range)	Install	Retain/install	Full gate
MEMIT-high	$8.3016\ [7.8805,8.7210]$	$2.0781\ [-3.2031,-1.1250]$	0.4781	1.0000	pass
ROME-low	$7.5166\ [7.0202,7.9752]$	$4.6406\ [-6.4219,-3.0781]$	0.4172	0.9925	inconclusive : locality

Appendix

Table 14: Disjoint new-160-family confirmation.

A

is query-time acquisition in nats,

D=-L

is terminal locality damage in percentage points, and brackets are frozen 95% empirical family-reweighting ranges.

Dose	$A$	95% range for $A$	$D$	Install / retain	Gates
1	6.37607	$[5.45820,7.23267]$	6.71875	0.3125 / 0.8000	A, R pass ; L, full inconclusive
2	6.37283	$[5.44759,7.24550]$	6.71875	0.3125 / 0.8000	A, R pass ; L, full inconclusive
4	6.28910	$[5.38021,7.19332]$	6.71875	0.3125 / 0.8000	A, R pass ; L, full inconclusive
8	5.60322	$[4.51207,6.67574]$	6.71875	0.3125 / 0.8000	A, R pass ; L, full inconclusive

Appendix

Table 15: Separately frozen ROME-low shared-eight-event facet.

A

R

, and

L

denote acquisition, retention, and locality;

A

is in nats and

D=-L

is terminal locality damage in percentage points. The locality endpoint and gate values are identical at all four doses because the full ordered event sequence and terminal state are identical. “Install / retain” reports installation and retention among installs.

Sensitivity target	Estimate (pp)	SE (pp)	95% range (pp)
Family-weighted point, relation-cluster CR1	$+2.5625$	0.9353	$[+0.6574,+4.4676]$
Equal weight over 33 observed relation means	$+2.3621$	0.9405	$[+0.4463,+4.2779]$

Appendix

Table 16: Post-hoc relation-aware sensitivity. Both ranges use a two-sided Student-

t

critical value with 32 degrees of freedom. CR1 denotes the cluster-robust finite-sample correction. Neither range replaces the frozen family-bootstrap primary; pp denotes percentage points.

Writer	Frozen prediction	Realized class	Acquisition range (nats)	Locality damage (pp)	Full-pass doses
AlphaEdit	safe + insensitive	safe + insensitive	$0.00122$ $[0,0.01301]$	$1.5625$	$4/4$
FT-L	failing + sensitive	failing + inconclusive	$0.61832$ $[0.08915,1.14749]$	$12.8125$	$0/4$

Appendix

Table 17: Additional-editor breadth on the shared eight-event construction. “Acquisition range” is the max–min difference across doses; brackets are its max-deviation-adjusted 95% reweighting band. “Safe” denotes this CounterFact neighborhood protocol only; pp denotes percentage points.

Figure 6: Observed editor contrasts vary across evaluated endpoint–bundle cells. Values are MEMIT-minus-ROME contrasts with separate pointwise paired-family ranges (160 families for CounterFact and zsRE; 128 for WikiData-CounterFact). Connecting lines only guide the eye across categorical endpoints. Because model, configuration, benchmark, and probe change together, the trajectories document heterogeneity rather than isolate its cause.

Context	MEMIT-high	ROME-low	Mean paired difference	Effect status
Canonical prompt	4.1261	3.9119	$+0.2142\ [+0.1079,+0.3369]$	pass
One rephrase	2.9104	2.1925	$+0.7178\ [+0.5530,+0.8903]$	pass

Appendix

Table 18: Prospectively frozen effect and transport record. Entries are mean dose-four-minus-reset per-target-token log probabilities in nats. Brackets are 95% empirical family-reweighting ranges from the fixed 64-family panel, not population confidence intervals.

Context	Policy	Acquisition	Install	Retain/inst.	Locality	Full
Canonical	MEMIT	4.1261	.8867	.9912	$-43.36\ [-52.73,-33.98]$	fail
Canonical	ROME	3.9119	.8047	.9563	$-41.41\ [-50.78,-32.23]$	fail
Rephrase	MEMIT	2.9104	.3828	1.0000	$-43.36\ [-52.73,-33.98]$	fail
Rephrase	ROME	2.1925	.2070	.9811	$-41.41\ [-50.78,-32.23]$	fail

Appendix

Table 19: Absolute feasibility fails despite transported mean acquisition. Acquisition is in nats; installation and retained-among-installs are proportions; locality is terminal-minus-reset relation-specificity success in percentage points. All entries are prospective.

Endpoint	Editor	$A$	$D$	MEMIT–ROME locality success
CounterFact	MEMIT-high	8.3016	2.0781	$+2.5625$ $[+1.0625,+4.2656]$
CounterFact	ROME-low	7.5166	4.6406	—
KnowEdit-zsRE	MEMIT-high	4.457329	31.328125	$-7.5000$ $[-11.015625,-4.218750]$
KnowEdit-zsRE	ROME-low	3.880083	23.828125	—
KnowEdit-WikiData-CF	MEMIT-high	3.997292	69.726562	$-15.9180$ $[-20.9961,-10.9375]$
KnowEdit-WikiData-CF	ROME-low	2.704324	53.808594	—

Appendix

Table 20: Fixed-strength GPT-2XL results under separately frozen endpoint-native instruments. Acquisition

A

is in nats and damage

D

in percentage points. The final column is the within-endpoint MEMIT-minus-ROME locality-success contrast with a paired-family 95% empirical reweighting range.

Endpoint	Families	Editor-level $L$ [95% range]	Paired $\Delta L$ [95% range]	Paired $\Delta A$ [95% range]
CounterFact	160	M: $-7.9531\ [-10.7188,-5.4688]$ R: $-29.1719\ [-33.4688,-25.0313]$	$+21.2188\ [+17.8438,+24.6094]$	$-0.3017\ [-0.5264,-0.1076]$
KnowEdit-zsRE	160	M: $-43.0469\ [-49.2969,-36.8750]$ R: $-47.5000\ [-53.7500,-41.3281]$	$+4.4531\ [+1.1719,+7.8125]$	$-0.2116\ [-0.3173,-0.1046]$
KnowEdit WikiData-CounterFact	128	M: $-70.5078\ [-75.7813,-65.1367]$ R: $-70.5078\ [-75.4883,-65.4297]$	$0.0000\ [-3.8086,+3.9063]$	$-0.2247\ [-0.4566,-0.0052]$

Appendix

Table 21: Frozen Llama-3.2-3B endpoint results. Editor-level signed locality (M: MEMIT; R: ROME) and paired M-minus-R contrasts.

L

is terminal-minus-reset success (points);

A

is acquisition (nats). Brackets are pointwise 95% empirical family-reweighting ranges (10,000 paired draws); ordering uses paired contrasts.

Record	Evidence	Status
Resolution	Outer-call reconstruction gives $s_{Z}=1$ for 640 CounterFact, 640 zsRE, and 512 WikiData-CF family–order pairs.	conditional pass ; diagnostic
Effect	Source acquisition/locality contrasts are $+0.7850$ $[+0.4416,+1.1441]$ nats and $+2.5625$ $[+1.0625,+4.2656]$ points; endpoint signs are $({+},{+},{+})$ / $({+},{-},{-})$ .	Acq. $H$ : undefined (no directional gate); locality $H$ : pass / fail / fail
Feasibility	CounterFact: MEMIT passes and ROME is inconclusive ; both fail locality on zsRE and retention/locality on WikiData-CF. Costs are ungated.	source mixed; targets fail
Transport	The correspondence fixes GPT-2XL, method identities, strengths, four-call template, dose, and orientation while endpoint data/instruments change. Locality reverses to $-7.5000$ $[-11.0156,-4.2188]$ and $-15.9180$ $[-20.9961,-10.9375]$ .	fail

Appendix

Table 22: Combined evidence for the fixed GPT-2XL comparison. Resolution is conditional on the outer-call adapter; cost has no declared gate;

H

is effect-decision status.

Endpoint	Locality success (points)	Acquisition (nats)
CounterFact	$+21.2188$ $[+17.2031,+25.4480]$	$-0.3017$ $[-0.5827,-0.0730]$
KnowEdit-zsRE	$+4.4531$ $[+0.3906,+8.5156]$	$-0.2116$ $[-0.3405,-0.0808]$
WikiData-CounterFact	$0.0000$ $[-4.6875,+4.8828]$	$-0.2247$ $[-0.5071,+0.0441]$

Appendix

Table 23: Bonferroni sensitivity. MEMIT-minus-ROME 98.333% marginal ranges span three endpoints; no population coverage is claimed.

Bundle	Endpoint	Acquisition (nats)	Locality (points)
GPT-2XL fixed	CounterFact	$+0.7850\ [+0.3779,+1.3489]$	$+2.5625\ [+0.5719,+4.9598]$
	zsRE	$+0.5772\ [+0.4149,+0.7682]$	$-7.5000\ [-12.0295,-3.2389]$
	WikiData-CF	$+1.2930\ [+1.0491,+1.5279]$	$-15.9180\ [-23.4375,-10.1958]$
Llama native	CounterFact	$-0.3017\ [-0.6196,-0.0818]$	$+21.2188\ [+16.9091,+25.5673]$
	zsRE	$-0.2116\ [-0.3327,-0.0931]$	$+4.4531\ [+0.4580,+8.7248]$
	WikiData-CF	$-0.2247\ [-0.4933,+0.1092]$	$0.0000\ [-5.4314,+6.5476]$

Appendix

Table 24: Post-hoc cluster-pairs sensitivity. MEMIT-minus-ROME points have Bonferroni-indexed 98.33% ranges within each model–metric family; CounterFact uses native IDs and KnowEdit exact templates.

Unsupported shortcut	Relevant evidence	Evaluation alternative
Equal update counts imply the same pre-answer history	24 schedule labels produce 1, 4, 12, or 24 history classes	Record the ordered updates before each answer
Immediate learning implies durable, local learning	LoRA: acquisition passes 6/6; retention and locality pass 0/6	Test acquisition, retention, and locality separately
A method ordering carries over to a new setting	Observed GPT-2XL and Llama-3.2-3B orderings differ across evaluated bundles; the cross-model mapping is incomplete	Evaluate the same methods directly in the new setting
An average improvement implies usability	WikiRecent mean improvement passes while locality fails	Require every prespecified behavioral check to pass

Appendix

Table 25: Four unsupported shortcuts and evidence-based alternatives.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Jul 13, 2026

PRISM Edit: One Vector for All Temporal Answers

Model editing keeps large language models (LLMs) up to date without retraining, but temporal facts expose a limitation of the prevailing locate-and-edit paradigm: an update is not always a replacement. When a fact changes, the new answer should become current while the old answer may remain correct in historical time contexts. Building on this insight, we use causal tracing to show that LLMs already support this distinction via a two-stage internal computation: early MLP layers retrieve a time-agnostic subject representation, and later layers modulate it with temporal context to yield the time-correct answer. Motivated by this finding, we introduce PRISM Edit, which optimizes a single polysemous representation across temporal contexts and leverages the model's inherent modulation pathway to route it to temporally correct predictions without requiring any architectural modification. We evaluate on TimeConflict, a newly introduced temporal editing benchmark, and on temporally augmented CounterFact. PRISM Edit improves multiple core metrics over the best baseline, most notably +23.3 Temporal Consistency (TC) and +33.7 Current Relative-time Score (CRS) on LLaMA-3, while being more than 2x faster. Code and data are publicly available at https://github.com/CheerCHuang/PRISM-Edit.

Chen Huang, Qi Zheng, Ruiqin Zheng +2

Tsinghua University · ByteDance

Temporal Reasoning in Language Models Model Editing Temporal Consistency Language Model Editing Temporal QA

Ask PDF

Sep 26, 2026

LLM Unlearning Evaluation with TRIAGE

Large language models can memorize private or harmful information, motivating machine unlearning methods that remove targeted knowledge while preserving other capabilities. However, existing evaluations rely primarily on behavioral benchmarks, which assess whether a model appears to forget but provide limited insight into how unlearning changes the model or affects related knowledge. We introduce TRIAGE (Tripartite Representation-internal Introspection for Adjacency Gap Evaluation), a benchmark-agnostic evaluation framework for characterizing these changes. TRIAGE uses diagonal approximations of the Fisher information and Hessian to measure changes in parameter sensitivity and local curvature, and utilizes a Forget / Adjacent-Retain / Generic-Retain partition to quantify an adjacency gap in semantically related knowledge. Based on the magnitude and distribution of these changes, TRIAGE further classifies each algorithm's update as no-op, partially localized, collateral dominant, or globally destructive. Across 12 unlearning methods, four language models, and the WMDP, TOFU, and MUSE benchmarks, we find that methods with similar behavioral forgetting can produce substantially different internal changes and patterns of collateral damage. These signatures also vary across models and benchmarks, indicating that the effects of unlearning are not determined solely by the unlearning algorithm. TRIAGE can be applied alongside existing unlearning benchmarks to complement behavioral evaluation with a model-internal view of how unlearning reshapes the model's parameter space and affects retained knowledge.

Danial Ataee, Peter Triantafillou

Department of Computer Science University of Warwick

Machine Unlearning Evaluation Machine Unlearning LLM Unlearning

Ask PDF

Oct 4, 2026

Self-Generated Feedback Destabilizes Test-Time Training: A Causal Decomposition of Long-Horizon Adaptation

Test-time training (TTT) lets a model store information in its weights during inference. When the model learns from its own output, however, each update also changes the model that generates the next training example. Across 128K-token streams, retaining generated-text updates worsens prediction on independent human-written text with three TTT-E2E model configurations (labeled 125M, 760M, and 3B). The same failure occurs when Adam updates Qwen3-4B's existing weights. The same update mechanisms can improve on real text, so writing itself is not the failure. Three matched comparisons trace the causal pathway. Fixed Generation removes over 98% of the damage at 125M and 760M by using a frozen model to generate training chunks. Recorded Replay separates the loss caused by reading degraded text from the additional loss stored by updating on it. A paired one-update comparison then shows the local conflict: an update predicts its source better but new real text worse. This cost grows after Closed Loop adaptation, with a few trajectories accounting for most large failures. Finally, Settlement evaluates the candidate state on independent real text before commitment. It leaves mean endpoint gaps of 0.07 and -0.02 nats at 125M and 760M while retaining real-text adaptation. These results motivate checking prediction on independent evidence before retaining an update.

Cheng Luo, Bing Li, Bernard Ghanem

King Abdullah University of Science and Technology (KAUST)

Test-Time Adaptation Continual Test-Time Adaptation Self-Training for Language Models Test-Time Training Inference-Time Adaptation of Language Models

Ask PDF

Abstract

LLM Evaluation

Explore similar work

PRISM Edit: One Vector for All Temporal Answers

LLM Unlearning Evaluation with TRIAGE

Self-Generated Feedback Destabilizes Test-Time Training: A Causal Decomposition of Long-Horizon Adaptation