cs.LGNov 4, 2026

The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention

Authors: Ye Yuan

Organizations: Independent Researcher

Abstract

Setting. This is an empirical characterization study; we propose no new method and run no comparisons against alternative objectives. We fine-tune Qwen3.5-4B with sparse LoRA and length-normalized DPO plus an SFT retention term (the RPO objective) on structured financial-filing tasks, and track downstream capability as the implicit reward margin grows to ∼7\sim 7 ββ-scaled units (nf4 training; ∼3\sim 3 under fp16 measurement on the unquantized base). Question. Does margin growth degrade capability uniformly across tasks, or selectively? Findings. In a pre-registered study (8 seeds, 6 unseen at freeze) with artifact-aware scoring and TOST equivalence testing, the alignment tax is graded and task-selective: extraction is preserved (Δ=−0.55Δ= -0.55 pts, equivalent at ±5\pm5 pts), json loses little (Δ=−3.52Δ= -3.52 pts, equivalence fails on a clustered item-level interval), and router degrades severely (Δ=−30Δ= -30 pts). Router's loss concentrates in a margin 1-3 damage window containing a transient format-break spike. The margin is accounted for almost entirely (≥96%\geq 96\%) by suppression of rejected completions, with chosen reward approximately flat, consistent with the retention term anchoring it; we do not attribute the behavioral damage to any single gradient channel. Gold-logp deflation is task-selective (extraction −0.15-0.15, json −1.04-1.04, router −1.47-1.47 nats/token under fp16), with gold selectively protected relative to incorrect controls on extraction (8/8 seeds). QLoRA reward margins differ up to 3.4×3.4\times between nf4 training and unquantized fp16 measurement, with an axis-dependent gap. Scope. Length-normalized rewards with SFT retention, sparse LoRA on 8/32 layers, one 4B model, easily-separable synthetic preferences, three structured tasks. A no-retention control and a measured format-alignment metric are left to future work.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

    May 12, 2026Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van +3Pairwise Preference LearningLLM Alignment