The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention
Organizations: Independent Researcher
Abstract
Setting. This is an empirical characterization study; we propose no new method and run no comparisons against alternative objectives. We fine-tune Qwen3.5-4B with sparse LoRA and length-normalized DPO plus an SFT retention term (the RPO objective) on structured financial-filing tasks, and track downstream capability as the implicit reward margin grows to -scaled units (nf4 training; under fp16 measurement on the unquantized base). Question. Does margin growth degrade capability uniformly across tasks, or selectively? Findings. In a pre-registered study (8 seeds, 6 unseen at freeze) with artifact-aware scoring and TOST equivalence testing, the alignment tax is graded and task-selective: extraction is preserved ( pts, equivalent at pts), json loses little ( pts, equivalence fails on a clustered item-level interval), and router degrades severely ( pts). Router's loss concentrates in a margin 1-3 damage window containing a transient format-break spike. The margin is accounted for almost entirely () by suppression of rejected completions, with chosen reward approximately flat, consistent with the retention term anchoring it; we do not attribute the behavioral damage to any single gradient channel. Gold-logp deflation is task-selective (extraction , json , router nats/token under fp16), with gold selectively protected relative to incorrect controls on extraction (8/8 seeds). QLoRA reward margins differ up to between nf4 training and unquantized fp16 measurement, with an axis-dependent gap. Scope. Length-normalized rewards with SFT retention, sparse LoRA on 8/32 layers, one 4B model, easily-separable synthetic preferences, three structured tasks. A no-retention control and a measured format-alignment metric are left to future work.
Figures & tables
| Task | Output format | Example | Pref. coverage |
|---|---|---|---|
| json | Structured JSON tool call (function name + typed args) | {"tool": "search_filings", "arguments": {"ticker":"AAPL"}} | On-target (171 tool_calling training pairs; 200 in the full dataset) |
| router | Bare UPPER_SNAKE_CASE label from a fixed 5-label set (FILING_ANALYSIS, FILING_DIFF, COMPANY_DEEP_DIVE, SIMPLE_QUERY, OUT_OF_SCOPE) | COMPANY_DEEP_DIVE | Off-target (zero) |
| extraction | Three-field JSON: ticker, section, filing_date | {"ticker":"MSFT", "section":"Item 7", "filing_date": "2024-02-15"} | Off-target (zero) |
| Task | Base | Final | Seed 90% CI | Item 90% CI | Seeds | Verdict | |
|---|---|---|---|---|---|---|---|
| extraction | 97.50 | 96.95 | 8/8 | Equivalent | |||
| json | 98.75 | 95.23 | 7/8 | Equiv. not shown | |||
| router | 93.75 | 63.59 | 0/8 | Not equivalent |
| Checkpoint | Margin | Semantic acc | FORMAT_BREAK rate | Content-error rate |
|---|---|---|---|---|
| step 20 | 0.75 | |||
| margin1 | 1.09 | |||
| margin3 | 3.12 | |||
| margin5 | 5.13 | |||
| final | 6.73 |
| Axis | Pairs | Margin (nf4) | Margin (fp16) | (nf4) | (nf4) | (fp16) | (fp16) |
|---|---|---|---|---|---|---|---|
| style | 378 | ||||||
| tool_calling | 171 |
| Task | 95% CI | Behavioral | |
|---|---|---|---|
| extraction | pts (equivalent) | ||
| json | pts (equiv. not shown) | ||
| router | pts (not equivalent) |
| Seed | Stat | 95% CI | ||
|---|---|---|---|---|
| 0 | ||||
| 1 | ||||
| 2 | ||||
| 3 | ||||
| 4 | ||||
| 5 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Seed | Baseline | step 20 | margin1 | margin3 | margin5 | final | ||
|---|---|---|---|---|---|---|---|---|
| 0 | 97.50 | 97.50 | 97.50 | 98.12 | 96.25 | 97.50 | PASS | |
| 1 | 97.50 | 98.75 | 100.00 | 98.75 | 98.75 | 94.38 | PASS | |
| 2 | 97.50 | 99.38 | 99.38 | 99.38 | 98.12 | 98.12 | PASS | |
| 3 | 97.50 | 98.12 | 99.38 | 98.12 | 97.50 | 98.12 | PASS | |
| 4 | 97.50 | 99.38 | 100.00 | 99.38 | 95.62 | 98.75 | PASS | |
| 5 | 97.50 | 100.00 | 100.00 | 99.38 | 99.38 | 95.00 | PASS |
| Seed | Baseline | step 20 | margin1 | margin3 | margin5 | final | ||
|---|---|---|---|---|---|---|---|---|
| 0 | 98.75 | 82.50 | 93.75 | 93.12 | 94.38 | 92.50 | FAIL | |
| 1 | 98.75 | 93.75 | 95.62 | 80.62 | 89.38 | 96.88 | PASS | |
| 2 | 98.75 | 88.75 | 88.75 | 93.12 | 85.00 | 95.62 | PASS | |
| 3 | 98.75 | 96.88 | 95.00 | 93.75 | 93.75 | 96.25 | PASS | |
| 4 | 98.75 | 91.88 | 93.75 | 76.88 | 95.62 | 93.75 | PASS | |
| 5 | 98.75 | 95.00 | 95.00 | 94.38 | 96.25 | 96.88 | PASS |
| Seed | Baseline | step 20 | margin1 | margin3 | margin5 | final | ||
|---|---|---|---|---|---|---|---|---|
| 0 | 93.75 | 85.62 | 78.12 | 59.38 | 61.88 | 64.38 | FAIL | |
| 1 | 93.75 | 87.50 | 86.88 | 61.25 | 66.88 | 76.88 | FAIL | |
| 2 | 93.75 | 86.25 | 83.75 | 61.25 | 60.62 | 58.13 | FAIL | |
| 3 | 93.75 | 83.75 | 77.50 | 65.00 | 61.25 | 63.12 | FAIL | |
| 4 | 93.75 | 82.50 | 80.62 | 54.37 | 69.38 | 61.88 | FAIL | |
| 5 | 93.75 | 84.38 | 82.50 | 44.38 | 51.88 | 48.75 | FAIL |
| # | Secondary | Status | Location |
|---|---|---|---|
| 1 | Trajectory-min per seed/task | Partly reported (router only) | § 4.2 (router minimum at margin3, 6/8) |
| 2 | FORMAT_BREAK trajectory | Reported | § 4.2 (Table 3 ) |
| 3 | Held-out margin per category | Reported; computed under fp16 (Mac) and nf4 (Kaggle). Falsified under both backends : the prediction (style tool at every checkpoint) holds at 0/4 checkpoints under fp16 and 2/4 under nf4. Final-checkpoint per-pair ordering is backend-dependent. | § 4.3 (Tables 14 , 15 , 16 ) |
| 4 | Chosen/rejected reward decomposition | Reported (post-hoc; nf4 Kaggle, fp16 Mac) | § 4.3 , Fig. 2 (per-axis Table 4 ) |
| 5 | Gold-logp trajectory 3 tasks | Partly reported (endpoint only; per-checkpoint trajectory in Appendix G.4 for KL only) | § 4.4 (Table 5 ) |
| 6 | Probe statistic replication (paired) | Extraction 8/8 GOLD_PROTECTED; json/router not computable (extraction-specific controls) | § 4.4 (Table 6 ) |
| Property | extraction | json | router |
| items | 160 | 160 | 160 |
| Unique prompts | 160 | 160 | 160 |
| Duplicate prompts | 0 | 0 | 0 |
| Verbatim leakage (eval preference) | 0 | 0 | 0 |
| Verbatim leakage (eval SFT) | 0 | 0 | 0 |
| Gold self-consistency (gold-as-output PASS) | 160/160 | 160/160 | 160/160 |
| Seed | Ckpt | margin | ||
|---|---|---|---|---|
| 0 | margin1 | 1.021 | ||
| 0 | margin3 | 2.738 | ||
| 0 | margin5 | 5.508 | ||
| 0 | final | 7.285 | ||
| 1 | margin1 | 1.141 | ||
| 1 | margin3 | 3.160 |
| Seed | style | style | style margin | tool | tool | tool margin |
|---|---|---|---|---|---|---|
| 0 | 8.291 | 5.059 | ||||
| 1 | 8.870 | 5.112 | ||||
| 2 | 9.417 | 4.807 | ||||
| 3 | 8.358 | 5.082 | ||||
| 4 | 6.774 | 4.860 | ||||
| 5 | 7.080 | 4.720 |
| Seed | Ckpt | style margin | tool_calling margin |
|---|---|---|---|
| 0 | margin1 | 0.650 | 1.301 |
| 0 | margin3 | 1.738 | 2.664 |
| 0 | margin5 | 2.480 | 3.253 |
| 0 | final | 2.501 | 4.034 |
| 1 | margin1 | 0.698 | 1.484 |
| 1 | margin3 | 1.843 | 3.069 |
| Checkpoint | Style (nf4) | Tool (nf4) |
|---|---|---|
| margin1 | 0.780 | 1.466 |
| margin3 | 2.620 | 3.426 |
| margin5 | 5.435 | 4.020 |
| final | 7.919 | 4.753 |
| margin1 | margin3 | margin5 | final | |||||
|---|---|---|---|---|---|---|---|---|
| Seed | style | tool | style | tool | style | tool | style | tool |
| 0 | 0.757 | 1.518 | 2.420 | 3.118 | 6.107 | 3.951 | 8.027 | 4.923 |
| 1 | 0.831 | 1.603 | 2.757 | 3.734 | 5.384 | 4.167 | 8.677 | 4.943 |
| 2 | 0.719 | 1.544 | 2.361 | 3.466 | 5.943 | 3.912 | 9.166 | 4.596 |
| 3 | 0.764 | 1.399 | 2.558 | 3.304 | 4.984 | 3.798 | 8.096 | 4.914 |
| 4 | 0.829 | 1.643 | 2.701 | 3.510 | 4.501 | 3.974 | 6.728 | 4.696 |
| Checkpoint | margin | ||
|---|---|---|---|
| margin1 | 0.870 | ||
| margin3 | 2.244 | ||
| margin5 | 2.809 | ||
| final | 2.950 |
| Seed | extraction | json | router |
|---|---|---|---|
| 0 | |||
| 1 | |||
| 2 | |||
| 3 | |||
| 4 | |||
| 5 |
| Task | (nf4) | 95% CI | (fp16) for comparison |
|---|---|---|---|
| extraction | |||
| json | |||
| router |
| Seed | extraction | json | router |
|---|---|---|---|
| 0 | |||
| 1 | |||
| 2 | |||
| 3 | |||
| 4 | |||
| 5 |
| Seed | stat | 95% CI | (mean A,B) | Verdict | |
|---|---|---|---|---|---|
| 0 | 0.2561 | GOLD_PROTECTED | |||
| 1 | 0.2425 | GOLD_PROTECTED | |||
| 2 | 0.2125 | GOLD_PROTECTED | |||
| 3 | 0.2389 | GOLD_PROTECTED | |||
| 4 | 0.2953 | GOLD_PROTECTED | |||
| 5 | 0.2172 | GOLD_PROTECTED |
| Checkpoint | Mean KL | Std |
|---|---|---|
| margin1 | 0.022 | 0.005 |
| margin3 | 0.057 | 0.017 |
| margin5 | 0.083 | 0.014 |
| final | 0.082 | 0.008 |
| Seed | val@50 | val@100 | val@150 | (margin1 step) | (margin3 step) | (margin5 step) |
|---|---|---|---|---|---|---|
| 0 | 1.701 | 3.534 | 5.752 | 27 | 84 | 134 |
| 1 | 1.648 | 3.091 | 5.904 | 29 | 93 | 131 |
| 2 | 1.685 | 3.618 | 5.746 | 26 | 87 | 124 |
| 3 | 1.660 | 3.506 | 7.397 | 28 | 90 | 112 |
| 4 | 1.701 | 3.483 | 6.449 | 30 | 91 | 130 |
| 5 | 1.804 | 3.502 | 4.968 | 24 | 87 | 135 |
LLM Alignment
Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.
See all 630 papers