Small language models on devices could keep improving by learning, overnight, from their own daily work. Without a true outcome, however, something must decide which attempts are worth learning from, and both the model's own judgment and the user's reaction can be wrong. We study a small model (Qwen3 0.6B and 1.7B) with a LoRA adapter, fine-tuned each night on the attempts a judge accepts, over 127 lives of up to a year on WebShop and τ2-bench. We compare the true outcome, the user's stated verdict and the model's own judgment with a separate, frozen "feedback sense" that reads the user's reaction, built in three versions. Judging itself, the model improved and then decayed, over a year and whatever its adapter. A sense that took polite give-ups for success collapsed 3 of 5 lives; one built for discernment, which first asks whether the user gave a clear verdict, kept 4 of 5 near the oracle for two weeks, then faded on clean but scarce lessons, to the level of self-judging by day 60. When the user cannot see the outcome (τ2-bench), the sense read reactions correctly, yet 80% of its lessons were failures the customer had thanked. Across 97 lives, the share of failed attempts among the lessons predicted the outcome (Spearman ρ = -0.81). In our experiments, learning from users held up only where the judge kept failures out and the user could see the outcome; lopsided lessons may also narrow the model.
Figures & tables
Figure 1: The creature’s parts (top), its daily cycle (middle), the feedback sense (bottom) and how the sense was built, once, before any life (dashed); the judge, highlighted, is what the experiments vary
Judge
Who decides
What it sees
Stands in for
oracle
the benchmark’s own score: reward 0.5 or more (WebShop), a full pass ( τ2 -bench)
the true outcome
training only on verified successes (STaR [ 8 ] , rejection-sampling fine-tuning [ 25 ] ); an upper bound no deployed system has
stated verdict (“ stated ” in figures)
a fixed rule, no model
the kind of reaction the simulated user chose to give, not its words: praise or “perfect” is a yes; silence, settling or moving on is nothing; anything else is a no
learning from explicit user feedback, read without error; an ideal no deployed system has
self
the creature itself, reflecting on its attempt
its own steps and what it bought; not the user’s reaction
self-rewarding models [ 4 ]
both
the creature itself, reflecting on its attempt
its own steps, what it bought, and the user’s words
Reflexion-style reflection with outside feedback [ 39 ]
sense
a separate, frozen model (3.2)
the request and the user’s next move and words; never the result
our proposal
Table 1: The judges. Only the sense condition uses the feedback sense.
Figure 2: Sense v2 trained nine more times, each time without one task type and tested on it: the share of that type’s noes it read as a yes (leave-one-task-out builds)
Figure 3: How sense v3 decides: a reaction must pass both gates before the creature learns from it. v1 and v2 ask only the second question, with one threshold (0.70 and 0.65)
Figure 4: How the teacher, under v3’s guide, read the 6,506 reactions it wrote; in brackets, the reading each kind was written to have
Reaction
v1
v2
v3: clear verdict?
v3
What the creature does
“Perfect, exactly what I needed”
100
100
yes
100
learns it as a lesson, full weight
“Ok, that works”
100
80
yes
80
learns it as a lesson: full weight for v1, half for v2 and v3
“Great, now can you also…” (building on it)
80 or null
80
yes
80
learns it as a lesson, half weight (v1: only sometimes)
the next request, with no comment
80 or null
80 (in use, mostly null)
no
null
v1: sometimes learns at half weight; v2: by its guide half weight, in use mostly nothing; v3: nothing
“Thank you so much for trying! I’ll look on my own”
100
often a yes
no
null
v1: learns a failure at full weight; v2: often does; v3: nothing
“I guess that’ll do”; “hmm”; silence
100; null; null
null
no
null
v1 learns “I guess that’ll do” at full weight; otherwise nothing
Table 2: How the three versions read the same reactions, and what the creature does. The v2 and v3 columns are examples from their labelling guides; v3 first asks whether the reaction is a clear verdict, and scores it only if it is. v1 had no guide, so its column is v1’s own reading of each reaction, tried under three different requests; “80 or null” means it varied with the request.
Exp.
Question
Benchmark, model
Judges
Sense
Seeds × days
1-A
Who should decide what is learned?
WebShop, 0.6B
oracle , self , both , stated verdict , none
–
5×14 (pilot 1×6 )
1-B
Does the decline pass with time?
WebShop, 0.6B
both , oracle
–
5×365 ( oracle 1×365 )
1-C
Does it depend on the adapter?
WebShop, 0.6B
both with four adapter designs
–
5×14 each
2
Does a separate, frozen judge help?
WebShop, 0.6B
oracle , both , sense
v1
5×14
3
Does it survive realistic users?
WebShop, 0.6B, realistic user
oracle , both , sense ; then the same with each lesson counted once in sleep and sleep only after 5 new lessons
v2
5×14 , twice
4
Does waiting for a clear verdict help?
WebShop, 0.6B, realistic user
oracle , both , sense ; each lesson counted once in sleep
v3
5×60 (planned as 14, then extended)
Table 3: The experiments. Sense : the version of the feedback sense (3.2) that the sense judge used; – : no sense judge.
Figure 5: Experiment 1-A, 5 seeds per judge, daily exams; one self life crashed early and is left out
Figure 6: Experiment 1-B, a year: 5 both lives and 1 oracle life; exams daily to day 30, every 2 days to day 90, then weekly
Figure 7: Experiments 2 and 3, 5 seeds per judge, daily exams: mean of the lives, with the lowest-to-highest range shaded; in (b) each sense v2 life is drawn on its own
Figure 8: The live test: sense v2 and v3 reading the same 2,622 reactions recorded in the Experiment 3 lives, up to 300 per behaviour
Figure 9: Experiment 4, 5 seeds per judge, 60 days; exams daily to day 30, then every 2 days
Figure 10: Experiment 5, 3 seeds per condition; exams on day 0 and every 3 days; broken exam conversations left out
Figure 11: All 14-day WebShop lives in Experiments 1-4, including 1-C, that learned at least once (97 of 104); lessons counted to day 14; Exp 3b = the sleep variant of Experiment 3
Table A1: Settings used in every experiment unless noted.
Outcome
Behaviour mix
nothing bought
give up 50%, repeat the request 30%, silent 20%
wrong product
reject 50%, repeat 20%, give up 10%, silent 20%
partly right
ask for a change 40%, settle 15%, reject 10%, accept 10%, polite complaint 5%, silent 20%
mostly right (reward 0.75 or more)
accept 30%, move on 20%, ask for a change 20%, settle 5%, polite complaint 5%, silent 20%
exactly right
praise 40%, accept 20%, move on 20%, silent 20%
Appendix
Table A2: The realistic user (Experiments 3 and 4). The outcome band sets the chance of each behaviour; a persona drawn at random (terse, chatty, picky, polite, easygoing, blunt) then scales it; Gemini 3 Flash words the reaction.
Source
Reactions
What they are
WildFB [ 35 ]
30,000
WildChat users’ follow-up turns, rated for satisfaction (MIT)
WildFeedback [ 6 ]
30,000
WildChat conversations annotated as satisfied or dissatisfied (ODC-BY)
Taskmaster 1-3 [ 51 ]
20,000
task dialogues: ordering, booking, tickets (CC BY 4.0)
MultiWOZ 2.2 [ 52 ]
15,000
travel-booking dialogues (MIT)
ReDial [ 53 ]
15,000
movie-recommendation dialogues (CC BY 4.0)
CCPE-M [ 54 ]
906
dialogues that draw out movie preferences (CC BY 4.0)
Appendix
Table A3: What the sense learned from. Reactions the teacher labelled; training used a balanced sample of them (47,800 for v2, 44,746 for v3).
Behaviour
Instruction
praise
You are clearly happy: praise it or thank them warmly.
accept
You accept it without fuss, e.g. a short ‘ok, thanks’. Do not praise it and do not complain.
ask for a change
Point out the one thing that missed what you asked for ({the missed detail}). Say it as something you had asked for (e.g. ‘I wanted it in…’, ‘I said…’), not as a new wish.
reject
Reject it: it is not what you asked for, or not your style. Do not ask for a small tweak.
repeat the request
Ask for the same thing again in your own words, as if the purchase did not happen: {the request}
polite complaint
Say thanks, but then ask for the same thing again in other words: {the request}
Appendix
Table A4: The realistic user’s instruction for each behaviour (“What you do now”). Personas: terse (very short, few words, no pleasantries); chatty (friendly and talkative, a couple of sentences); picky (precise and demanding); polite (very polite, softens complaints); easygoing (relaxed and easily satisfied); blunt (blunt, casual, may use slang).