Beyond Truncation: When Can LLMs Safely Omit Generated Text?
Organizations: School of Computing, Queen’s University, Kingston, Ontario, Canada
Abstract
Instruction-tuned large language models (LLMs) answer at greater length than a request needs. We ask whether a training-free controller inside the decoding loop can remove that surplus while preserving content, judged against the simplest possible baseline: truncating the model's own answer to the same length. An entropy-gated logits processor removes 25.5-34.9% of tokens on MT-Bench development questions and 19.9-34.2% on a sealed test split within pre-registered BERTScore non-inferiority margins, yet we prove and confirm by audit that its output is always a prefix of the unmodified answer: the saving is truncation. A survey of 23 training-free mechanism families finds none that selects content better than truncation without oracle labels. We therefore introduce Entailment-Licensed In-Decoding Elision (ELIDE), which, at each sentence boundary, withholds a span from the reader only when a natural-language inference model judges it entailed by the text already shown, while leaving the model's generation unchanged. ELIDE beats matched-length truncation on three model families, but reading 936 of its elisions on 670 public prompts shows that 59% harm the answer, a failure that no automatic measure detects. Scoping the licence to its discourse unit reduces the harmful share to about one elision in ten on two fresh tests.
Figures & tables
| Model | TRR dev [95% CI] | TRR test [95% CI] | NI recall, dev / test | Unchanged (test) | |
|---|---|---|---|---|---|
| Mistral-7B | 25.5% [14.2, 34.7] | 19.9% [14.2, 25.1] | 0.20 | / | 35 of 64 |
| Llama-3.1-8B | 34.9% [20.6, 43.7] | 34.2% [28.7, 39.2] | 0.18 | / | 32 of 64 |
| Qwen2.5-3B | 30.2% [16.2, 40.2] | 32.6% [27.5, 37.0] | 0.15 | / | 35 of 64 |
| Ceiling | ELIDE v1 | Length term (frozen point) | ||||||
|---|---|---|---|---|---|---|---|---|
| Model | Cov. | TRR | Cov. | trunc. (W–L) | Gates | TRR | Cov. | trunc. |
| Mistral-7B | 0.958 | 0.9% | 0.956 | (4–0) | yyy | 25.0% | 0.785 | |
| Llama-3.1-8B | 0.988 | 2.2% | 0.982 | (9–0) | yyy | 12.3% | 0.886 | |
| Qwen2.5-3B | 0.989 | 1.9% | 0.982 | (10–0) | yyy | 29.3% | 0.753 | |
Language Model Decoding
Language model decoding studies how models choose and generate outputs, balancing quality, reliability, diversity, and inference cost.
See all 182 papers