Researchers have explored different ways to improve large language models (LLMs)' capabilities via dummy token insertion in contexts. However, existing works focus solely on the dummy tokens themselves, but fail to leverage the inherent sentence-level structure of natural language. This is a critical oversight, as LLMs acquire linguistic capabilities through exposure to human-generated texts, which are inherently structured at the sentence level. Motivated by this gap, we propose an approach that inserts delimiters at sentence boundaries in LLM inputs, which not only integrates dummy tokens into the context, but also facilitates LLMs with sentence-by-sentence processing behavior during reasoning. Two concrete methods: (1). In-context learning and (2). Supervised fine-tuning are experimented using 7B models to 600B Deepseek-V3. Our results demonstrate consistent improvements across various tasks, with notable gains of up to 7.7% on GSM8k and 12.5% on DROP. Furthermore, the fine-tuned LLMs can incorporate sentence awareness evidenced by their internal representations. Our work establishes a simple yet effective technique for enhancing LLM's capabilities, offering promising directions for cognitive-inspired LLM enhancement paradigm.
Figures & tables
Figure 1: Overview of Sentence-Level Inference: We insert delimiters at sentence boundaries to enable LLMs to “pause and integrate context” during inference. Two approaches are proposed: (1) In-Context Learning (ICL): LLMs infer with delimiter placement from exemplars in long contexts; (2) Supervised Fine-Tuning (SFT): LLMs learn sentence-segmented patterns via delimiter-inserted training data. For Llama3-8B-Instruct, this approach improves performance by ∼ 4.4% on GSM8k and ∼ 6.8% on DROP over unsegmented inputs.
Figure 2: The distributions of sentence lengths and number of sentences for each dataset. The left column figures are the origin distribution, and the right column figures are zoomed-in views. Horizontal bars indicate medians and extrema. Sentence lengths are counted by number of tokens, from the Llama3 tokenizer.
Qwen2-7B-Inst
Llama3-8B-Inst
Qwen2.5-72B-Inst
Deepseek-V3
Dataset
base
seg
Δ
base
seg
Δ
base
seg
Δ
base
seg
Δ
MMLU
64.43
69.96
+5.53% ↑
62.89
67.28
+4.39% ↑
86.64
86.40
-0.24% ↓
74.04
74.82
+0.78% ↑
GSM8k
73.92
81.65
+7.73% ↑
75.51
78.01
+2.5% ↑
90.14
91.96
+1.82% ↑
95.00
95.30
+0.3% ↑
MATH
53.33
54.30
+0.97% ↑
32.60
32.26
-0.34% ↓
73.04
75.78
+2.74% ↑
89.40
90.60
+1.2% ↑
DROP
38.14
50.64
+12.50% ↑
46.39
53.16
+6.77% ↑
58.74
60.38
+1.64% ↑
75.10
79.10
+4% ↑
Table 1: In-Context Learning results. We compare the performance of vanilla inference (base) against ICL (seg), delimiter here is “<seg>”. Δ denotes the absolute improvement. Our method yields consistent gains across models and tasks, with particularly strong improvements on smaller models and in reading comprehension task.
MMLU
GSM8k
MATH
DROP
MMLU-pro
GPQA
HumanEval
Std-FT
59.02
72.48
30.86
48.50
34.25
26.93
56.71
Pause-FT
56.11
75.44
33.50
55.97
35.71
24.16
-
Seg-FT
60.13
74.91
31.58
54.26
40.71
27.43
62.80
Table 2: Supervised Fine-Tuning results on LLaMA3-8B-Base. Our method (Seg-FT) is compared against standard fine-tuning (Std-FT) and pause-token fine-tuning (Pause-FT). Best performance is in bold , and results outperforming the Std-FT baseline are underlined . Our approach demonstrates superior robustness and generalization.
Figure 3: Performance of different delimiter choices in ICL across three datasets. More structured delimiters could consistently yield a better performance, demonstrating the value of a clear, non-semantic structural signal. “orig.” denotes the baseline without any delimiters.
Figure 4: Sentence segmentation (Sent) vs. fixed n-token chunking. Sentence-level segmentation consistently outperforms fixed-chunking strategies, whose effectiveness decrease when the chunk size ( n ) is either too large or too small, only peaking when n is close to the majority sentence length.
Figure 5: Sentence-level vs. random delimiter placement. Meaningful placement at sentence boundaries contributes more to the performance gains, far surpassing the minor effect of random insertions.
Std-FT
Seg-FT
Improvement
Prob
61.90
61.19
-0.71%
CoT
59.02
60.13
+1.12%
Table 3: MMLU zero-shot performance of SFT models under two evaluation protocols. The benefits of our method manifest exclusively in the CoT setting, highlighting its role in enhancing deliberative reasoning.
Figure 6: Relative attention scores for different delimiter types on the GSM8k dataset. Our delimiter (Sent. delimiter) receives significantly higher attention than both the sentence average (N × larger than avg.) and traditional punctuation delimiters (punc. delimiter).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
GSM8k
DROP
no-seg
71.42
50.90
seg
74.91
54.26
Appendix
Table 4: Comparison between SFT models with segmented input (seg) and raw input (no-seg).
Figure 7: Attention map of Llama3-8b-seg
Figure 8: Attention map of Qwen2-7b-Instruct. The segmentation token we used is “ ####” . We replaced it to “ <seg> ” only when visualization.
Structured prompting organizes instructions, context, reasoning steps, or output schemas to make language-model behavior more reliable and controllable.
Contextual entrainment, which is a newly discovered phenomenon in large language models (LLMs), refers to the tendency of a model to assign higher probabilities to tokens that appear in its context. In this work, we extend this phenomenon from the token level to the sentence level by examining the per-token mean log-probability of a sentence instead of the probabilities of individual tokens. We investigate sentence-level contextual entrainment across 26 LLMs from seven families and two datasets, which cover both subjective and objective tasks. We find that sentence-level contextual entrainment exists. This means that the sentences in the prompt (even if they are counterfactual statements) can significantly increase their probability during model inference time. As the model size increases, contextual entrainment gradually decreases. We also find that contextual entrainment is controlled by 2% to 4% of the attention heads. Turning off these attention heads can effectively mitigate contextual entrainment without hurting the model's performance.
Transformer-based large language models (LLMs) such as RoBERTa represent text using contextual word embeddings (CWEs), which alter the embeddings associated with each token based on surrounding context. We construct token-wise incremental trajectories by repeatedly recomputing a token's CWE as successive words are added to a sentence, yielding a representation of how contextualized embeddings evolve as the utterance unfolds. We evaluate this approach using garden-path sentences as a test case with characteristic features. Token-wise trajectories reproduce known features of garden-path processing, including disruption around the critical region, and reliably distinguish garden-path sentences from matched disambiguated controls. We introduce several metrics for quantifying representational displacement across contextual increments and show that trajectory information can be highly predictive of sentence type. We find that ambiguity-related information is recoverable not only from the sentence-level CLS representation but also from ordinary vocabulary tokens, suggesting that utterance-level information is distributed across multiple representational scales. In exploratory analyses, we find qualitatively similar trajectory structures in other ambiguity- and misdirection-related linguistic phenomena. Together, these results establish token-wise incremental trajectories as a promising framework for studying utterance-specific meaning construction using LLMs.
Grayson Wycliffe Storer, Julia Witte Zimmerman
Department of Mathematics and Statistics. · Department of Computer Science, University of Vermont, Burlington, 05405, Vermont, United States. · Computational Story Lab. +1
Large language models are typically trained with next-token prediction (NTP), which rewards predicting specific tokens rather than explicitly accounting for the semantic similarities between different words or expressions. Prior work has addressed this limitation by augmenting training data with minimally varying sentences that replace a target word with conceptually related alternatives, e.g., "The tired coach ran home" → {"The tired trainer ran home", "The tired instructor ran home", ...}. Such concept augmentation improves performance, but it remains unclear whether its effectiveness depends on where in the sentence the concept is expressed. We reproduce and extend prior work beyond nouns to verbs and adjectives, asking whether the part of speech (POS) carrying the concept signal affects perplexity and downstream performance. We find different patterns across evaluation settings: concept augmentation improves perplexity most consistently on news, independent of the POS used, whereas verbs provide the most reliable downstream gains. Despite these behavioral differences, representational analyses using CKA and linear probing show that the conditions yielding the largest downstream gains are not necessarily those that most alter the model's internal representations. Together, these results suggest that the effectiveness of concept augmentation depends on where and how conceptual information is introduced during training.