Grounding Language Models in Speech Does Not Fix "AI Flavor": Calibrated LLM Judges, Goodhart Traps, and a Preference-Tuning Fix
Organizations: Independent Researcher
Abstract
Chinese outputs of large language models carry a perceptible "AI flavor": formulaic sentence frames, stacked written-register connectives, repetition loops, and layout leakage. A popular hypothesis blames text-only training for the missing inner-speech layer of human writing; injecting speech or prosody signals should therefore reduce it. We test it with controlled interventions on a 0.6B model; all three fail: (i) phoneme-interleaved continued pretraining yields no spokenness gain under text-matched controls while raising repetition (+0.13 [+0.03, +0.23]); (ii) prosody-plan conditional generation reaches an adherence of only 0.257 against a pre-registered 0.50 gate; (iii) a rule-based reward built on TTS prosody-feature distance is imperceptible in double-blind evaluation by both humans (8/16) and a calibrated LLM judge (9/16). A root cause is evaluation: single-rater blind evaluation at n~20 only detects extreme (>=85%) preferences. We therefore build a calibrated LLM-judge protocol: the judge passes catch trials on known-answer pairs (16/16; single-text 22/24, same-topic pairs 24/24, versus 0.75 for the human rater), then annotates with two independent judges under A/B swapping, counting disagreement as a tie. Using the judge's agree labels directly as DPO pairs (no reward model), we obtain the first positive result: held-out blind win rate 0.833 [0.699, 0.972], repetition 0.114 to 0.000, unchanged MMLU. We further identify a Goodhart channel: the judge's length preference transduces into output-length collapse via pairwise DPO (median 75 chars vs a 300-char prompt); an SFT-anchor repairs it (w is a style knob: w=0.1 preserves gains, w=0.3 does not). Scaling preference data from 89 to 240 pairs yields no additional judge-visible gain. We release RA-Eval v0.1: a three-layer Chinese AI-flavor evaluation framework with 382 judge-annotated preference pairs.
Figures & tables
| Model | Data | Recipe | Median chars | Held-out blind win rate |
|---|---|---|---|---|
| base | – | – | 237 | – |
| iter-1 (v1) | 89 | DPO 300 | 151 | |
| iter-2 | 240 | DPO 480 | 75 | rejected: length collapse |
| iter-3 | 90 ( ) | DPO 270 | 121 | rejected: still short |
| iter-4 | 240 | DPO+SFT | 173 | rejected: style washed out |
| iter-5 (v5) | 240 | DPO+SFT | 156 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Base | Qwen3-0.6B (instruct) 50M-token text-only CPT (P1a noplan arm) DPO |
|---|---|
| Training | LoRA ( ) on all projections, fp32 |
| DPO | lr , , batch 1, accum 1, cosine to 0, seq 1024 |
| SFT anchor | , |
| Data | 240 agree pairs (train) / 67 (held-out); ; not both degenerate |
| Judge | GLM-5.3-Flash, isolated sub-agents, A/B swap double-pass, disagreement |
Language Modeling
Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.
See all 547 papers