Think in Sentences: Explicit Sentence Boundaries Enhance Language Model's Capabilities

Zhichen Liu; Yongyuan Li; Yang Xu

cs.CLApr 11, 2026

Think in Sentences: Explicit Sentence Boundaries Enhance Language Model's Capabilities

Authors: Zhichen Liu, Yongyuan Li, Yang Xu

Organizations: Department of Computer Science and Engineering Southern University of Science and Technology

Abstract

Researchers have explored different ways to improve large language models (LLMs)' capabilities via dummy token insertion in contexts. However, existing works focus solely on the dummy tokens themselves, but fail to leverage the inherent sentence-level structure of natural language. This is a critical oversight, as LLMs acquire linguistic capabilities through exposure to human-generated texts, which are inherently structured at the sentence level. Motivated by this gap, we propose an approach that inserts delimiters at sentence boundaries in LLM inputs, which not only integrates dummy tokens into the context, but also facilitates LLMs with sentence-by-sentence processing behavior during reasoning. Two concrete methods: (1). In-context learning and (2). Supervised fine-tuning are experimented using 7B models to 600B Deepseek-V3. Our results demonstrate consistent improvements across various tasks, with notable gains of up to 7.7% on GSM8k and 12.5% on DROP. Furthermore, the fine-tuned LLMs can incorporate sentence awareness evidenced by their internal representations. Our work establishes a simple yet effective technique for enhancing LLM's capabilities, offering promising directions for cognitive-inspired LLM enhancement paradigm.

Figures & tables

Figure 1: Overview of Sentence-Level Inference: We insert delimiters at sentence boundaries to enable LLMs to “pause and integrate context” during inference. Two approaches are proposed: (1) In-Context Learning (ICL): LLMs infer with delimiter placement from exemplars in long contexts; (2) Supervised Fine-Tuning (SFT): LLMs learn sentence-segmented patterns via delimiter-inserted training data. For Llama3-8B-Instruct, this approach improves performance by $\sim$ 4.4% on GSM8k and $\sim$ 6.8% on DROP over unsegmented inputs.

Figure 2: The distributions of sentence lengths and number of sentences for each dataset. The left column figures are the origin distribution, and the right column figures are zoomed-in views. Horizontal bars indicate medians and extrema. Sentence lengths are counted by number of tokens, from the Llama3 tokenizer.

	Qwen2-7B-Inst			Llama3-8B-Inst			Qwen2.5-72B-Inst			Deepseek-V3
Dataset	base	seg	$\Delta$	base	seg	$\Delta$	base	seg	$\Delta$	base	seg	$\Delta$
MMLU	64.43	69.96	+5.53% $\uparrow$	62.89	67.28	+4.39% $\uparrow$	86.64	86.40	-0.24% $\downarrow$	74.04	74.82	+0.78% $\uparrow$
GSM8k	73.92	81.65	+7.73% $\uparrow$	75.51	78.01	+2.5% $\uparrow$	90.14	91.96	+1.82% $\uparrow$	95.00	95.30	+0.3% $\uparrow$
MATH	53.33	54.30	+0.97% $\uparrow$	32.60	32.26	-0.34% $\downarrow$	73.04	75.78	+2.74% $\uparrow$	89.40	90.60	+1.2% $\uparrow$
DROP	38.14	50.64	+12.50% $\uparrow$	46.39	53.16	+6.77% $\uparrow$	58.74	60.38	+1.64% $\uparrow$	75.10	79.10	+4% $\uparrow$

Table 1: In-Context Learning results. We compare the performance of vanilla inference (base) against ICL (seg), delimiter here is “<seg>”.

\Delta

denotes the absolute improvement. Our method yields consistent gains across models and tasks, with particularly strong improvements on smaller models and in reading comprehension task.

	MMLU	GSM8k	MATH	DROP	MMLU-pro	GPQA	HumanEval
Std-FT	59.02	72.48	30.86	48.50	34.25	26.93	56.71
Pause-FT	56.11	75.44	33.50	55.97	35.71	24.16	-
Seg-FT	60.13	74.91	31.58	54.26	40.71	27.43	62.80

Table 2: Supervised Fine-Tuning results on LLaMA3-8B-Base. Our method (Seg-FT) is compared against standard fine-tuning (Std-FT) and pause-token fine-tuning (Pause-FT). Best performance is in bold , and results outperforming the Std-FT baseline are underlined . Our approach demonstrates superior robustness and generalization.

Figure 3: Performance of different delimiter choices in ICL across three datasets. More structured delimiters could consistently yield a better performance, demonstrating the value of a clear, non-semantic structural signal. “orig.” denotes the baseline without any delimiters.

Figure 4: Sentence segmentation (Sent) vs. fixed n-token chunking. Sentence-level segmentation consistently outperforms fixed-chunking strategies, whose effectiveness decrease when the chunk size (

n

) is either too large or too small, only peaking when

n

is close to the majority sentence length.

Figure 5: Sentence-level vs. random delimiter placement. Meaningful placement at sentence boundaries contributes more to the performance gains, far surpassing the minor effect of random insertions.

	Std-FT	Seg-FT	Improvement
Prob	61.90	61.19	-0.71%
CoT	59.02	60.13	+1.12%

Table 3: MMLU zero-shot performance of SFT models under two evaluation protocols. The benefits of our method manifest exclusively in the CoT setting, highlighting its role in enhancing deliberative reasoning.

Figure 6: Relative attention scores for different delimiter types on the GSM8k dataset. Our delimiter (Sent. delimiter) receives significantly higher attention than both the sentence average (N

\times

larger than avg.) and traditional punctuation delimiters (punc. delimiter).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	GSM8k	DROP
no-seg	71.42	50.90
seg	74.91	54.26

Appendix

Table 4: Comparison between SFT models with segmented input (seg) and raw input (no-seg).

Figure 7: Attention map of Llama3-8b-seg

Figure 8: Attention map of Qwen2-7b-Instruct. The segmentation token we used is “ ####” . We replaced it to “ <seg> ” only when visualization.

Structured Prompting

Structured prompting organizes instructions, context, reasoning steps, or output schemas to make language-model behavior more reliable and controllable.

See all 22 papers

Sentence-Level Contextual Entrainment in Large Language Models
Jun 23, 2026Yang Liu, Chenhui ChuLLM Interpretability
Kyoto University
Contextual trajectory and incremental contextual displacement: Towards using LLMs to understand dynamic, utterance-specific meaning construction
Sep 30, 2026Grayson Wycliffe Storer, Julia Witte ZimmermanText Embeddings Language Modeling
Department of Mathematics and Statistics. · Department of Computer Science, University of Vermont, Burlington, 05405, Vermont, United States. · Computational Story Lab. +1
Every Part Has Its Role: Part-of-Speech Effects in LLM Post-Training
Oct 21, 2026Laya Iyer, Chen ShaniLanguage Model Post-Training Knowledge Augmentation for Language Models
Stanford University · Tel Aviv University

Jun 23, 2026J/K move · Enter open · S save

Sentence-Level Contextual Entrainment in Large Language Models

Yang Liu, Chenhui Chu

Kyoto University

LLM Interpretability

Open paper AskPDF arXiv

Contextual entrainment, which is a newly discovered phenomenon in large language models (LLMs), refers to the tendency of a model to assign higher probabilities to tokens that appear in its context. In this work, we extend this phenomenon from the token level to the sentence level by examining the per-token mean log-probability of a sentence instead of the probabilities of individual tokens. We investigate sentence-level contextual entrainment across 26 LLMs from seven families and two datasets, which cover both subjective and objective tasks. We find that sentence-level contextual entrainment exists. This means that the sentences in the prompt (even if they are counterfactual statements) can significantly increase their probability during model inference time. As the model size increases, contextual entrainment gradually decreases. We also find that contextual entrainment is controlled by 2% to 4% of the attention heads. Turning off these attention heads can effectively mitigate contextual entrainment without hurting the model's performance.

Abstract

Structured Prompting

Explore similar work

Sentence-Level Contextual Entrainment in Large Language Models

Contextual trajectory and incremental contextual displacement: Towards using LLMs to understand dynamic, utterance-specific meaning construction

Every Part Has Its Role: Part-of-Speech Effects in LLM Post-Training