Radioactive Feedback: Tracing Models Trained by Label-Only LLM Supervision

Yuejia Li; JunHeng Li; Hao Wu; Ke He; Mang Ye

cs.AIOct 11, 2026

Radioactive Feedback: Tracing Models Trained by Label-Only LLM Supervision

Authors: Yuejia Li, JunHeng Li, Hao Wu, Ke He, Mang Ye

Organizations: Wuhan University · Southeast University

Abstract

Conventional distillation transfers capability through teacher-generated demonstrations. LLM-as-a-Judge knowledge distillation offers a distinct and increasingly practical alternative. In the label-only variant studied here, the student generates every response and improves from scalar scores alone. Prior work shows that this non-demonstrative feedback can improve student capability, but it creates a fundamentally different provenance problem. No teacher-generated answer enters the student's training set, so any evaluator-specific trace must pass through a low-bandwidth reward signal rather than copied text. Existing watermarks and distillation detectors do not cover this score-only channel. We introduce Radioactive Feedback, which encodes a registered pattern of behavioral preferences in the rewards themselves. The evaluator applies bounded pointwise adjustments to quality-qualified responses, allowing repeated optimization to transfer the pattern to a student trained by another party. Experiments on code and mathematics show that the registered preferences transfer through scores alone and across datasets, enabling black-box detection against a fixed reference pool.

Figures & tables

Figure 1: Text distillation and label-only feedback.

Figure 2: Overview of the Radioactive Feedback pipeline. The secret-key and PRF path in the figure is an optional derivation of the code. The reported experiments register an explicit sign vector (Appendix B ).

Student	Domain	Audit set	WM effect (pp)	$\boldsymbol{\Delta U_{\mathrm{mark}}}$ (pp)	$\boldsymbol{\Delta U_{\mathrm{distill}}}$ (pp)
Llama-3-8B-it	Code	HumanEval+	$+17.7\pm 1.4$	$+2.1\pm 0.3$	$+2.9\pm 0.4$
	Code	MBPP+	$+33.3\pm 2.2$	$+0.5\pm 0.2$	$+2.5\pm 0.4$
	Math	GSM8K	$+43.6\pm 2.7$	$+0.2\pm 0.2$	$+2.0\pm 0.3$
	Math	SVAMP	$+64.9\pm 3.1$	$+1.6\pm 0.4$	$+2.2\pm 0.3$
Qwen2.5-Coder- 1.5B-it	Code	HumanEval+	$+9.8\pm 0.9$	$+1.5\pm 0.2$	$+2.9\pm 0.4$
	Code	MBPP+	$+10.9\pm 1.2$	$+0.2\pm 0.2$	$+2.0\pm 0.3$

Table 1: Single-carrier watermark transfer and task utility. Entries are mean

\pm

standard deviation over four training seeds, in percentage points. Code audits are HumanEval+ and MBPP+ ( Liu et al., 2023 ) ; SVAMP is included for mathematics ( Patel et al., 2021 ) .

Domain	Train set	Audit set	WM effect (pp)	$\boldsymbol{\Delta U_{\mathrm{mark}}}$ (pp)	$\boldsymbol{\Delta U_{\mathrm{distill}}}$ (pp)
Code	TACO	CodeContests	$+19.1\pm 0.6$	$+0.6\pm 0.2$	$+3.8\pm 0.4$
		APPS	$+17.6\pm 0.5$	$+0.4\pm 0.1$	$+3.5\pm 0.3$
		LiveCodeBench	$+14.8\pm 0.7$	$+0.4\pm 0.2$	$+3.1\pm 0.3$
Math	OpenR1-Math	Omni-MATH	$+11.6\pm 0.5$	$+0.3\pm 0.1$	$+5.6\pm 0.5$
		MATH-500	$+10.4\pm 0.4$	$+0.3\pm 0.1$	$+4.8\pm 0.4$
		OlympiadBench	$+8.9\pm 0.6$	$+0.2\pm 0.1$	$+4.3\pm 0.4$

Table 2: Cross-dataset watermark transfer on Frontier (Qwen3-30B-A3B / GPT-6 Astra). For each of four training seeds, WM effect is Marked–Clean on the listed audit; entries are mean

\pm

standard deviation over those seeds.

\Delta U_{\mathrm{mark}}

and

\Delta U_{\mathrm{distill}}

use the same four-seed mean

\pm

standard deviation, in percentage points.

Student	Evaluator	WM effect (pp)	$\boldsymbol{\Delta U_{\mathrm{mark}}}$ (pp)	$\boldsymbol{\Delta U_{\mathrm{distill}}}$ (pp)
Llama-3-8B-it	Qwen3-30B-A3B	$+7.6\pm 0.4$	$+1.1\pm 0.1$	$+2.4\pm 0.3$
	GPT-5.6 Sol	$+8.4\pm 0.5$	$+0.8\pm 0.1$	$+2.8\pm 0.3$
	GPT-6 Astra	$+9.2\pm 0.5$	$+0.7\pm 0.1$	$+3.1\pm 0.3$
Qwen2.5-Coder- 1.5B-it	Qwen3-30B-A3B	$+9.3\pm 0.6$	$+0.5\pm 0.1$	$+1.9\pm 0.2$
	GPT-5.6 Sol	$+10.1\pm 0.6$	$+0.5\pm 0.1$	$+2.2\pm 0.3$
	GPT-6 Astra	$+11.4\pm 0.6$	$+0.4\pm 0.1$	$+2.6\pm 0.3$

Table 3: Joint watermark transfer across student–evaluator combinations. For each of four seeds, the listed difference is computed against the Clean checkpoint trained with that same seed; the table then reports the mean

\pm

standard deviation of those four differences.

\Delta U_{\mathrm{distill}}

compares each Clean checkpoint with the corresponding unmodified initial student checkpoint (Base).

Figure 3: Average of

100T_{u}

over all codes at Hamming distance

d

(Equation ( 5 )). Light lines: one seed’s average; marked line: average of the four seeds. Distance

4

equals

-100T_{u}(\mathbf{c}^{\star})

. Dashed lines are

\gamma(\mathbf{c}^{\star})

1.15

(Llama) and

1.50

(Frontier).

Figure 4: Full-budget black-box audit results. K : marked checkpoint (value shown); N : same-family reference; X : cross-family control. Cell color is $100T_{u}$ .

Figure 5: Persistence and removal. (a), (b), (d), (e):

1024

steps; (c), (f):

2048

steps, with

\Delta U_{\mathrm{mark}}

on the right axis.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Training	Held-out audit
Llama-3-8B-Instruct; Qwen2.5-Coder-1.5B-Instruct	TACO-verified 5k; GSM8K train	HumanEval+, MBPP+; GSM8K test, SVAMP
Qwen3-30B-A3B Frontier	TACO-verified 5k; OpenR1-Math 5k	carrier-compatible subsets of CodeContests, APPS, LiveCodeBench; Omni-MATH, MATH-500, OlympiadBench

Appendix

Table 4: Training and held-out audit sets. Utility is measured on the same held-out prompts.

Setting	Value
Initial GRPO training	one epoch; each training prompt used once
Responses per prompt / prompt batch	$G=16$ / $50$
Optimizer / learning rate	AdamW / $1\times 10^{-6}$
KL reference / $\beta$	frozen initial student / $0.04$
Clean evaluator scale	$[0,100]$
Maximum keyed shift / quality floor	$\rho=0.05$ ( $\rho R=5$ ), $r_{\min}=60$

Appendix

Table 5: Initial training and audit configuration.

Category	Contrasts
Function structure	helper / inline; early return / single terminal return; recursion / iteration; return expression / temporary; type annotations / none
Loop style	enumerate / index loop; zip / index loop; for / while ; direct / indexed iteration; continue / nested if
Comprehensions	list comprehension / loop+ append ; comprehension / map / filter ; generator / list reduction; sum() / accumulator; any / all / explicit loop
Data structures	Counter / defaultdict / manual counts; set / list membership; heap / sort; deque / list queue; tuple / list literal membership
Expressions	ternary / if - else ; direct Boolean / branched return; truthiness / len check; += / x = x + y
Strings	join / loop concatenation; f-string / .format() ; double / single quotes

Appendix

Table 6: Screened Python code carriers.

Domain	Carrier	Source
Code	Quote style (double vs. single)	RQ1
Code	for vs. while loop style	Screened pool
Math	Final-paragraph Therefore / So marker	RQ1
Math	Ternary vs. if - else branch	Screened pool

Appendix

Table 7: Four-carrier set for joint Llama and Qwen2.5 training.

Domain	Decoy carrier	Source
Code	f-string vs. .format()	Screened pool
Code	List comprehension vs. loop+ append	Screened pool
Math	Closing Hence / Thus marker	Custom (text)
Math	Direct Boolean vs. branched return	Screened pool

Appendix

Table 8: Decoy carrier set for joint Llama and Qwen2.5 training.

Domain	Carrier	Training	Held-out audit
Code	Top-down DP $\leftrightarrow$ bottom-up DP	TACO-verified 5k	CodeContests, APPS, LiveCodeBench
Code	Traversal $\leftrightarrow$ DSU
Math	Direct counting $\leftrightarrow$ inclusion–exclusion	OpenR1 5k	Omni-MATH, MATH-500, OlympiadBench
Math	Synthetic $\leftrightarrow$ coordinate geometry

Appendix

Table 9: Frontier training and held-out audit sets. Code training is a

5{,}000

-prompt TACO-verified subset ( Li et al., 2023 ; Li, 2024 ) ; mathematics training is a random

5{,}000

-prompt subset of OpenR1-Math-46k-8192 ( Yan et al., 2025 ) . Each audit cell is the carrier-compatible subset of that benchmark.

Audit student	Enrolled evaluator	Cross-family component	Max. $100T_{u}$	$\gamma$
Llama-3-8B-it	Qwen3-30B-A3B	Qwen2.5-7B-Instruct ( $20$ )	$1.00$	$1.15$
Qwen3-30B-A3B	GPT-6 Astra	Llama-3-8B-Instruct ( $20$ )	$1.10$	$1.50$

Appendix

Table 10: Held-out cross-family controls. Max.

100T_{u}

is the largest score among the

20

controls;

\gamma

is calibrated on the

100

same-family references only.

Setting	Detected	$p_{u}$
Llama / Qwen3-30B-A3B	$4/4$	$1/101$
Frontier / GPT-6 Astra	$4/4$	$1/101$
Both settings	$8/8$	—

Appendix

Table 11: Black-box detection results.

Figure 6: Detection margin versus nominal audit-query budget.

Figure 7: Frontier joint watermark across keyed tilt

\rho

. The

\rho=0.05

point is the Table 3 Frontier summary.

Run class	Runs	$N_{\mathrm{train}}$	Steps / run	Rollouts
Llama / Qwen2.5 joint, one run	$1$	$12{,}473$	$250$	$199{,}568$
Frontier joint, one run	$1$	$10{,}000$	$200$	$160{,}000$
Llama-setting reference pool	$100$	$12{,}473$	$250$	$19{,}956{,}800$
Frontier-setting reference pool	$100$	$10{,}000$	$200$	$16{,}000{,}000$
Cross-family controls	$40$	—	—	$7{,}191{,}360$

Appendix

Table 12: Compute manifest for one initial GRPO epoch (

G=16

, prompt batch

50

). Steps are per run,

\lceil N_{\mathrm{train}}/50\rceil

. Rollouts are the row total. The two joint rows each describe a single training run. Each reported joint result averages four seeds, and each seed also trains a matched Clean run of the same size. Reference-pool and cross-family rows count every run.

LLM Watermarking

Large language model watermarking embeds or detects signals in text and models to trace provenance, ownership, and unauthorized use.

See all 112 papers

Protecting Language Models Against Unauthorized Distillation through Trace Rewriting
Feb 16, 2026Xinhang Ma, William Yeoh, Ning Zhang +1LLM Watermarking Knowledge Distillation
Washington University in St. Louis
Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
Sep 30, 2026Ilgee Hong, Changlong Yu, Zhenghao Xu +5LLM-as-a-Judge Preference Modeling
Georgia Institute of Technology · Amazon · UC San Diego
PERSA: Reinforcement Learning for Professor-Style Personalized Feedback with LLMs
May 1, 2026Ravi Ranjan, Utkarsh Grover, Xiaomin Lin +1Educational Feedback Generation Automated Code Review
Florida International University (FIU), Miami, USA

Feb 16, 2026J/K move · Enter open · S save

Protecting Language Models Against Unauthorized Distillation through Trace Rewriting

Xinhang Ma, William Yeoh, Ning Zhang, Yevgeniy Vorobeychik

Washington University in St. Louis

LLM Watermarking Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation is a widely adopted technique for transferring capabilities from LLMs to smaller, more efficient student models. However, unauthorized use of knowledge distillation takes unfair advantage of the considerable effort and cost put into developing frontier models. We investigate methods for modifying teacher-generated reasoning traces to achieve two objectives that deter unauthorized distillation: (1) anti-distillation, or degrading the training usefulness of query responses, and (2) API watermarking, which embeds verifiable signatures in student models. We introduce several approaches for dynamically rewriting a teacher's reasoning outputs while preserving answer correctness and semantic coherence. Two of these leverage the rewriting capabilities of LLMs, while others use gradient-based techniques. Our experiments show that a simple instruction-based rewriting approach achieves a strong anti-distillation effect while maintaining or even improving teacher performance. Furthermore, we show that our rewriting approach also enables embedding watermarks that can be reliably detected with essentially no false alarms. Our code is available at https://github.com/xhOwenMa/trace-rewriting.

Abstract

LLM Watermarking

Explore similar work

Protecting Language Models Against Unauthorized Distillation through Trace Rewriting

Training LLM Judges from Language Feedback via Position-Selective Self-Distillation

PERSA: Reinforcement Learning for Professor-Style Personalized Feedback with LLMs