Small language models on devices could keep improving by learning, overnight, from their own daily work. Without a true outcome, however, something must decide which attempts are worth learning from, and both the model's own judgment and the user's reaction can be wrong. We study a small model (Qwen3 0.6B and 1.7B) with a LoRA adapter, fine-tuned each night on the attempts a judge accepts, over 127 lives of up to a year on WebShop and τ2-bench. We compare the true outcome, the user's stated verdict and the model's own judgment with a separate, frozen "feedback sense" that reads the user's reaction, built in three versions. Judging itself, the model improved and then decayed, over a year and whatever its adapter. A sense that took polite give-ups for success collapsed 3 of 5 lives; one built for discernment, which first asks whether the user gave a clear verdict, kept 4 of 5 near the oracle for two weeks, then faded on clean but scarce lessons, to the level of self-judging by day 60. When the user cannot see the outcome (τ2-bench), the sense read reactions correctly, yet 80% of its lessons were failures the customer had thanked. Across 97 lives, the share of failed attempts among the lessons predicted the outcome (Spearman ρ = -0.81). In our experiments, learning from users held up only where the judge kept failures out and the user could see the outcome; lopsided lessons may also narrow the model.
Figures & tables
Figure 1: The creature’s parts (top), its daily cycle (middle), the feedback sense (bottom) and how the sense was built, once, before any life (dashed); the judge, highlighted, is what the experiments vary
Judge
Who decides
What it sees
Stands in for
oracle
the benchmark’s own score: reward 0.5 or more (WebShop), a full pass ( τ2 -bench)
the true outcome
training only on verified successes (STaR [ 8 ] , rejection-sampling fine-tuning [ 25 ] ); an upper bound no deployed system has
stated verdict (“ stated ” in figures)
a fixed rule, no model
the kind of reaction the simulated user chose to give, not its words: praise or “perfect” is a yes; silence, settling or moving on is nothing; anything else is a no
learning from explicit user feedback, read without error; an ideal no deployed system has
self
the creature itself, reflecting on its attempt
its own steps and what it bought; not the user’s reaction
self-rewarding models [ 4 ]
both
the creature itself, reflecting on its attempt
its own steps, what it bought, and the user’s words
Reflexion-style reflection with outside feedback [ 39 ]
sense
a separate, frozen model (3.2)
the request and the user’s next move and words; never the result
our proposal
Table 1: The judges. Only the sense condition uses the feedback sense.
Figure 2: Sense v2 trained nine more times, each time without one task type and tested on it: the share of that type’s noes it read as a yes (leave-one-task-out builds)
Figure 3: How sense v3 decides: a reaction must pass both gates before the creature learns from it. v1 and v2 ask only the second question, with one threshold (0.70 and 0.65)
Figure 4: How the teacher, under v3’s guide, read the 6,506 reactions it wrote; in brackets, the reading each kind was written to have
Reaction
v1
v2
v3: clear verdict?
v3
What the creature does
“Perfect, exactly what I needed”
100
100
yes
100
learns it as a lesson, full weight
“Ok, that works”
100
80
yes
80
learns it as a lesson: full weight for v1, half for v2 and v3
“Great, now can you also…” (building on it)
80 or null
80
yes
80
learns it as a lesson, half weight (v1: only sometimes)
the next request, with no comment
80 or null
80 (in use, mostly null)
no
null
v1: sometimes learns at half weight; v2: by its guide half weight, in use mostly nothing; v3: nothing
“Thank you so much for trying! I’ll look on my own”
100
often a yes
no
null
v1: learns a failure at full weight; v2: often does; v3: nothing
“I guess that’ll do”; “hmm”; silence
100; null; null
null
no
null
v1 learns “I guess that’ll do” at full weight; otherwise nothing
Table 2: How the three versions read the same reactions, and what the creature does. The v2 and v3 columns are examples from their labelling guides; v3 first asks whether the reaction is a clear verdict, and scores it only if it is. v1 had no guide, so its column is v1’s own reading of each reaction, tried under three different requests; “80 or null” means it varied with the request.
Exp.
Question
Benchmark, model
Judges
Sense
Seeds × days
1-A
Who should decide what is learned?
WebShop, 0.6B
oracle , self , both , stated verdict , none
–
5×14 (pilot 1×6 )
1-B
Does the decline pass with time?
WebShop, 0.6B
both , oracle
–
5×365 ( oracle 1×365 )
1-C
Does it depend on the adapter?
WebShop, 0.6B
both with four adapter designs
–
5×14 each
2
Does a separate, frozen judge help?
WebShop, 0.6B
oracle , both , sense
v1
5×14
3
Does it survive realistic users?
WebShop, 0.6B, realistic user
oracle , both , sense ; then the same with each lesson counted once in sleep and sleep only after 5 new lessons
v2
5×14 , twice
4
Does waiting for a clear verdict help?
WebShop, 0.6B, realistic user
oracle , both , sense ; each lesson counted once in sleep
v3
5×60 (planned as 14, then extended)
Table 3: The experiments. Sense : the version of the feedback sense (3.2) that the sense judge used; – : no sense judge.
Figure 5: Experiment 1-A, 5 seeds per judge, daily exams; one self life crashed early and is left out
Figure 6: Experiment 1-B, a year: 5 both lives and 1 oracle life; exams daily to day 30, every 2 days to day 90, then weekly
Figure 7: Experiments 2 and 3, 5 seeds per judge, daily exams: mean of the lives, with the lowest-to-highest range shaded; in (b) each sense v2 life is drawn on its own
Figure 8: The live test: sense v2 and v3 reading the same 2,622 reactions recorded in the Experiment 3 lives, up to 300 per behaviour
Figure 9: Experiment 4, 5 seeds per judge, 60 days; exams daily to day 30, then every 2 days
Figure 10: Experiment 5, 3 seeds per condition; exams on day 0 and every 3 days; broken exam conversations left out
Figure 11: All 14-day WebShop lives in Experiments 1-4, including 1-C, that learned at least once (97 of 104); lessons counted to day 14; Exp 3b = the sleep variant of Experiment 3
Table A1: Settings used in every experiment unless noted.
Outcome
Behaviour mix
nothing bought
give up 50%, repeat the request 30%, silent 20%
wrong product
reject 50%, repeat 20%, give up 10%, silent 20%
partly right
ask for a change 40%, settle 15%, reject 10%, accept 10%, polite complaint 5%, silent 20%
mostly right (reward 0.75 or more)
accept 30%, move on 20%, ask for a change 20%, settle 5%, polite complaint 5%, silent 20%
exactly right
praise 40%, accept 20%, move on 20%, silent 20%
Appendix
Table A2: The realistic user (Experiments 3 and 4). The outcome band sets the chance of each behaviour; a persona drawn at random (terse, chatty, picky, polite, easygoing, blunt) then scales it; Gemini 3 Flash words the reaction.
Source
Reactions
What they are
WildFB [ 35 ]
30,000
WildChat users’ follow-up turns, rated for satisfaction (MIT)
WildFeedback [ 6 ]
30,000
WildChat conversations annotated as satisfied or dissatisfied (ODC-BY)
Taskmaster 1-3 [ 51 ]
20,000
task dialogues: ordering, booking, tickets (CC BY 4.0)
MultiWOZ 2.2 [ 52 ]
15,000
travel-booking dialogues (MIT)
ReDial [ 53 ]
15,000
movie-recommendation dialogues (CC BY 4.0)
CCPE-M [ 54 ]
906
dialogues that draw out movie preferences (CC BY 4.0)
Appendix
Table A3: What the sense learned from. Reactions the teacher labelled; training used a balanced sample of them (47,800 for v2, 44,746 for v3).
Behaviour
Instruction
praise
You are clearly happy: praise it or thank them warmly.
accept
You accept it without fuss, e.g. a short ‘ok, thanks’. Do not praise it and do not complain.
ask for a change
Point out the one thing that missed what you asked for ({the missed detail}). Say it as something you had asked for (e.g. ‘I wanted it in…’, ‘I said…’), not as a new wish.
reject
Reject it: it is not what you asked for, or not your style. Do not ask for a small tweak.
repeat the request
Ask for the same thing again in your own words, as if the purchase did not happen: {the request}
polite complaint
Say thanks, but then ask for the same thing again in other words: {the request}
Appendix
Table A4: The realistic user’s instruction for each behaviour (“What you do now”). Personas: terse (very short, few words, no pleasantries); chatty (friendly and talkative, a couple of sentences); picky (precise and demanding); polite (very polite, softens complaints); easygoing (relaxed and easily satisfied); blunt (blunt, casual, may use slang).
Small language models often fail explicit user requests (length, number of items, format, required or forbidden words, grammatical person, answer language). We measure a training-free governance layer placed outside the model: it reads the requests with rules, checks the answer mechanically, repairs it when a request is not kept, and hands back the original answer when the repair fails. The loop exists in prior work; our contribution is measuring its benefits, costs and failures, and reporting how the measuring instrument itself was calibrated. On four 4-bit Qwen-family models (3B, 7B, 14B, one 7B reasoning model) and 329 Chinese prompts: (1) answers that keep every request the layer read rise significantly (all comparisons p<0.0001), by 34 to 66 of 329 under the corrected checker and 52 to 92 after a length-control fix; a generic redo message brings no significant gain. (2) Exploratory: on 90 repair-triggered prompts, two AI annotators found the layer produced an acceptable answer where the bare model did not more often than the reverse (49:7; 18:12, not significant). (3) Costs: 43-66% more generation time; whole-answer rewriting makes a sizeable share of usable answers unusable. (4) Failures: rule-based reading reaches about 70% and 84% of a two-annotator ceiling on unseen phrasings; unsatisfiable requests make the reasoning model loop; model self-reports are unreliable (18 of 25 "fixed" items were not fixed). (5) Over half of the original checker's "violated" verdicts on structural requests were wrong; after correction the main result holds, and one comparison is downgraded. Conclusions are limited to this Chinese material, the Qwen family and sizes up to 14B. The governance implementation is not released; data, answers, verdicts and the checker are public.
We set out to train behavioral dispositions (self-verification, uncertainty acknowledgment, feedback integration) into small language models (0.6B to 2.3B effective parameters) through a four-stage all-MIT distillation pipeline, with follow-on experiments on inference-time attention-head interventions and a frozen-base confidence-gated sidecar. An internal draft reported +33.9-point MCAS and +15.3-point HumanEval gains on a Qwen3-0.6B student; a second-pass sanity check falsified both numbers before publication. The HumanEval delta was a truncation artifact (n_predict=512) that inverted to -8.0 points at n_predict=1024; the MCAS gain disappeared under apples-to-apples scoring. That falsification triggered three subsequent arcs. Across (1) SFT/DPO LoRA on three model families and two domains, (2) inference-time attention-head tempering on o_proj, and (3) a training-free frozen-base sidecar reading the final-token hidden state h_last, we find no operator that moves judge-measured disposition without damaging content or collapsing into stylistic mimicry. The failure is consistent across five models (Qwen3-0.6B, Qwen3-1.7B, Qwen3.5-0.8B, Gemma 4 E2B, and SmolLM2-1.7B-Instruct). A within-distribution cross-validation pass (AUC=0.683) collapsed to chance on fresh prompts (AUC=0.516). We contribute a three-arc negative result with mechanism, a two-failure-mode taxonomy for linear h_last probes, and an honest falsification pipeline that converts the class of false positives we ourselves produced into publishable negatives. As an independent finding, Gemma 4 E2B exhibits near-complete confidence-correctness decoupling on the Chef domain (assertion asymmetry -0.009; the model asserts at 91% regardless of correctness).
Language models often abandon a correct answer when a user pushes back without giving a reason, and this behavior is usually studied as sycophancy to be suppressed. The same pushback, however, also leads models to revisit and sometimes repair a wrong answer. We measure both effects on multiple-choice questions from MMLU and ARC-Challenge with four open models from two families (Qwen3 1.7B, 4B and 8B, and OLMo-2 7B), reading the final answer exactly from option logits after a free-form reply. The reply budget changes the verdict. When Qwen3-4B may answer the challenge in at most 96 tokens, pushback leaves accuracy unchanged or slightly lower; with 1,024 tokens, it corrects 43% of its wrong answers and pushback raises accuracy by 5.7 points. Whether pushback helps overall follows a simple break-even rule in the cave and correction rates. Inside the model, caving can be partly predicted before the challenge: hidden states add about 0.04-0.06 AUROC over turn-1 confidence in three of our four probing runs. A probe trained only on self-correction cases predicts caving beyond confidence when replies are truncated, but not when the model can finish its reasoning. Activation steering never reduced caving selectively: where it worked, it suppressed self-correction as well. We conclude that sycophancy evaluations should allow full replies and report self-correction alongside caving.
Ook Lee
Department of Information System, Hanyang University, Seoul, Korea