Attention-Aligned Reasoning for Large Language Models

Hongxiang Zhang; Yuan Tian; Tianyi Zhang

cs.CLOct 3, 2025

Attention-Aligned Reasoning for Large Language Models

Authors: Hongxiang Zhang, Yuan Tian, Tianyi Zhang

Organizations: Purdue University

Abstract

Large Language Models (LLMs) tend to generate a long reasoning chain when solving complex tasks. However, as the reasoning chain extends, critical intermediate steps and the original prompt will be buried in the context, receiving insufficient attention and leading to errors. In this work, we present ATAR, a novel reasoning method that leverages the inherent reasoning structure to steer LLM attention. Our experiments show that ATAR outperforms SOTA methods across six benchmarks, achieving up to 15.39% absolute improvement. Furthermore, with ATAR, "non-reasoning" models achieve comparable or even better performance compared to reasoning models of the same size in most benchmarks. Finally, our ablation studies show that the attention alignment component contributes significantly, and that these improvements are persist under different attentionsteering backends.

Figures & tables

Figure 1: Overview of Atar . By interleaving planning and action steps, our method explicitly steers attention to balance the global objective (the original question) and the local objective (the current plan step).

Figure 2: An example in Tree-of-Thought. ToT successfully generates a correct plan. The corresponding action is separated from the plan by a large amount of intervening context, thus deviating from the plan’s instruction.

Model	Method	Math			CommonSense		BBH
Model	Method	GSM8K	AQuA	MATH-500	StrQA	T4D	BBH
Llama3.2-3B	CoT	73.16	46.06	40.80	66.64	32.98	35.09
	PS+	76.95 (+3.79)	38.58 (-7.48)	42.20 (+1.40)	62.79 (-3.85)	31.81 (-1.17)	40.51 (+5.42)
	RE2	74.68 (+1.52)	47.28 (+1.22)	41.00 (+0.20)	65.03 (-1.61)	30.32 (-2.66)	38.02 (+2.93)
	ToT	70.89 (-2.27)	35.04 (-11.02)	32.80 (-8.00)	60.44 (-6.20)	40.80 (+7.82)	41.32 (+6.23)
	Atar	78.32 (+5.16)	48.43 (+2.37)	42.40 (+1.60)	67.55 (+0.91)	43.79 (+10.81)	50.48 (+15.39)

Table 1: Evaluation results on six benchmarks. Best results are shown in bold , and those indicating a performance gain compared to Chain-of-Thought are shown in green .

Model	BBH
Model	DQA	CJ	DU	LD	STED	Snarks
Llama3.2-3B	52.10	54.01	52.40	40.40	38.40	74.47
Phi4-mini	60.80	62.03	69.60	51.20	56.00	79.78
Qwen3-4B	74.00	61.50	77.20	89.60	64.40	84.27
Llama3.1-8B	60.80	53.48	64.00	51.20	52.80	71.35
Phi-4	70.40	65.24	87.20	81.20	59.20	90.45

Table 2: Performance of Atar across BBH task categories. Full task names are provided in Sec. 3.1 .

Model	Math			CommonSense		BBH
Model	GSM8K	AQuA	MATH-500	StrQA	T4D	BBH
Phi-4-mini-reasoning	85.37	67.97	87.80	66.38	45.04	59.85
Phi-4-mini (w/ Atar )	88.02	68.50	62.60	68.69	49.47	62.42
Qwen3-4B-Thinking	92.27	67.32	93.20	68.31	73.40	75.34
Qwen3-4B (w/ Atar )	93.56	79.92	88.60	70.13	71.56	75.31
Phi-4-reasoning	91.20	83.20	91.20	77.61	74.11	74.98

Table 3: Evaluation comparison with reasoning models within the same family and size.

Method	Math	CommonSense	BBH
CoT	24.17	57.59	31.67
PS+	23.66	55.02	32.89
RE2	24.96	58.38	33.60
ToT	22.56	49.46	28.74
Atar (w/ SPA)	29.17	59.77	35.01
Atar (w/ PASTA)	26.53	60.43	33.71

Table 4: Performance of Atar under different backends, including SPA ( Tian and Zhang, 2025 ) , and PASTA ( Zhang et al., 2024a ) .

Model	Method	Math			CommonSense		BBH
Model	Method	GSM8K	AQuA	MATH-500	StrQA	T4D	BBH
Llama3.1-8B	Atar (w/o attention steering)	79.01	53.15	47.60	71.17	35.28	52.77
Llama3.1-8B	Atar (w/o dynamic mechanism)	80.13	48.03	48.80	73.18	38.17	53.64
	Atar	83.93	55.51	50.20	73.54	40.01	58.53
Phi-4-mini-4B	Atar (w/o attention steering)	78.77	58.66	55.40	68.60	38.83	57.44
Phi-4-mini-4B	Atar (w/o dynamic mechanism)	84.32	62.60	57.20	67.97	48.47	60.80

Table 5: Ablation study comparing Atar with and without attention steering. Results show that attention steering consistently improves performance. Best results are highlighted in bold .

Panel of Figure 3 — Figure 3: Analysis of chain length and task complexity.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	AQuA-RAT	T4D
Llama3.1-8B	CoT	50.79	26.77
	Atar (Plan-only)	40.94	26.06
	Atar (Plan history)	53.15	36.70
	Periodic summarization (K=2)	46.06	39.82
	Atar ( Ours )	55.51	40.01
Phi-4-mini	CoT	61.81	39.54

Appendix

Table 6: Alternative design on anchoring strategies. Results show accuracy (%) on AQuA-RAT and T4D benchmarks.

Model	Method	AQuA-RAT	T4D
Llama3.1-8B	CoT	50.79	26.77
	Atar (Fixed Strength)	47.64	36.17
	Atar (Entropy)	47.64	35.64
	Atar (Arithmetic)	54.72	35.99
	Atar (Geometric)	55.11	35.28
	Atar (Harmonic)	55.51	40.01

Appendix

Table 7: Alternative design on confidence aggregation methods. “Fixed Strength" uses a constant anchor weight, while others calculate dynamic weights based on the mean token probability. Results show accuracy (%).

Model	Latency (Second/Task) $\downarrow$
Model	CoT	PS+	RE2	ToT	Atar
Llama3.2-3B	34.96	35.20 (+0.24)	27.64 (-7.32)	193.55 (+158.59)	48.06 (+13.10)
Phi-4-mini-4B	26.26	21.58 (-4.68)	24.27 (-1.99)	278.70 (+252.44)	42.30 (+16.04)
Qwen3-4B	61.24	89.97 (+28.73)	50.40 (-10.84)	292.28 (+231.04)	74.29 (+13.05)
Llama3.1-8B	109.82	110.54 (+0.72)	101.31 (-8.51)	428.2 (+318.38)	89.71 (-20.11)
Phi-4-15B	108.44	108.69 (+0.25)	108.66 (+0.22)	482.48 (+374.04)	122.59 (+14.15)

Appendix

Table 8: Efficiency comparison

TASK INSTRUCTIONS

You are an expert problem solver. Your task is to decompose the given problem into a clear, step-by-step plan, reasoning the plan, and solve the problem step by step in JSON format.

For each plan step, provide a key-value pair: the key is the plan step (as stated), the value is the detailed reasoning and action for that step.

Now, implement a reasoning structure to follow step-by-step and arrive at correct answers in JSON format.

TASK

Original problem: {question}

Appendix

Table 9: Prompt used by Atar to elicit structured planning and step-wise reasoning in format.

Chain-of-Thought

Let’s think step by step.

Re-Reading.

{question}

Read the question again:

{question}

Appendix

Table 10: Prompt used by Chain-of-Thought and Re-Reading.

Mathematical Reasoning Tasks

Let’s first understand the problem, extract relevant variables and their corresponding numerals, and make and devise a complete plan. Then, let’s carry out the plan, calculate intermediate variables (pay attention to correct numerical calculation and commonsense), solve the problem step by step, and show the answer.

COMMONSENSE TASKS

Let’s first prepare relevant information and make a plan. Then, let’s answer the question step by step (pay attention to commonsense and logical coherence).

Appendix

Table 11: Prompts used for the Plan-and-Solve+ baseline under different task settings.

Prompt	Question: A machine is set up in such a way that it will short circuit if both the black wire and the red wire touch the battery
	at the same time. The machine will not short-circuit if just one of these wires touches the battery. The black wire is designated
	as the one that is supposed to touch the battery, while the red wire is supposed to remain in some other part of the machine.
	One day, the black wire and the red wire both end up touching the battery at the same time. There is a short circuit. Did the
	black wire cause the short circuit?
	Options: (A) Yes (B) No
Atar	{ "Step 1: Understand the scenario": "The machine is designed to short circuit only when both the black wire

Appendix

Table 12: An example of BBH causal reasoning. Incorrect causal inference is highlighted in red .

Prompt	Question: Anna is able to buy 5 more articles for $300 after the price of each article decreased by 15%.
	What is the new selling price of each article?
	Options: (A) $8 (B)$ 10 (C) $13.6 (D)$ 22.9 (E) $40
Atar	{ "Step 1: Establish variables": "Let the original price of each article be $x$ .",
	"Step 2: Apply price decrease": "After a 15% decrease, the new price becomes $0.85x$ .",
	"Step 3: Set up the cost equation": "Anna buys 5 more articles for $300, so the total cost is represented as
	6 $\times 0.85x=300$ .",

Appendix

Table 13: An example of mathematical reasoning in Aqua. Incorrect numerical reasoning is highlighted in red .

Prompt	Sentence: The secretary came to the analyst’s office and helped her to book a flight.
Prompt	Options: (A) Helped the secretary book a flight (B) Helped the analyst book a flight (C) Ambiguous
Atar	{ "Step 1: Identify the pronoun": "The pronoun is ’her’.",
(w/o attention steering)	"Step 2: Determine the antecedent of the pronoun": " The antecedent of ’her’ is the subject of the sentence, which is ’The secretary’ ",
	"Step 3: Analyze the sentence structure": The subject of the sentence is ’The secretary’, and the verb is ’came’. The object of the
	verb ’came’ is ’to the analyst’s office’. The verb ’helped’ has the pronoun ’her’ as its object.",
	"Step 4: Determine the antecedent of the pronoun based on the sentence structure": "Based on the sentence structure, the antecedent

Appendix

Table 14: An example of BBH. Incorrect inferences are highlighted in red . Correct ones are highlighted in green .

Question:	Is the language used in Saint Vincent and the Grenadines rooted in English?
Chain-of-Thought (CoT)	No, the official language of Saint Vincent and the Grenadines is English, but the local Creole language,
	Saint Vincentian Creole, is also widely spoken. Saint Vincentian Creole is a English-based Creole language that
	incorporates elements of French, and African languages. It is the mother tongue of the majority of the population
	and is used in everyday conversation, music, and literature. While English is the official language and is used in
	government, education, and formal settings, Saint Vincentian Creole is the language of the people and is an
	important part of the country’s cultural identity. So, while English is the official language, the local Creole

Appendix

Table 15: An example illustrating Chain-of-Thought’s reasoning process.

Question:	Is the language used in Saint Vincent and the Grenadines rooted in English?
Plan-and-Solve (PS+)	Step 1: Understand the question
	The question asks about the language used in Saint Vincent and the Grenadines being rooted in English.
	This means we need to determine the primary language spoken in the country and its linguistic origins.
	Step 2: Research
	To answer this question, we need to research the languages spoken in Saint Vincent and the Grenadines. Saint
	Vincent and the Grenadines is an island nation in the southern part of the Windward Islands in the Caribbean Sea.

Appendix

Table 16: An example illustrating Plan-and-Solve’s reasoning process.

Question:	Is the language used in Saint Vincent and the Grenadines rooted in English?
Re-Reading (RE2)	No, the language used in Saint Vincent and the Grenadines is primarily English, but it also includes
	elements of French, African, and Indian languages due to its colonial history and diverse population.
	Final answer: No

Appendix

Table 17: An example illustrating Re-Reading’s reasoning process.

Question:	Is the language used in Saint Vincent and the Grenadines rooted in English?
Atar	{"Understand the question.": "
	The question is asking whether the language spoken in Saint Vincent and the Grenadines is based on English.",
	"Research the official language of Saint Vincent and the Grenadines.":
	"Saint Vincent and the Grenadines is an island nation in the Caribbean, and its official language is English.",
	"Confirm the linguistic roots of the official language.":
	"English is a Germanic language, which means it has roots in Old English, a language spoken in parts of what is now England."}

Appendix

Table 18: An example illustrating Atar ’s reasoning process.

Model	Method	disambiguation_qa	causal_judgement	date_understanding	logical_deduction	salient_translation_error_detection	snarks
Model	Method	BBH
Llama3.2-3B	CoT	88	71	102	34	95	89
	PS+	104	63	97	86	98	105
	re-read	74	84	127	40	89	105
	ToT	81	90	120	73	93	107
	Atar	131	101	131	101	96	129

Appendix

Table 19: BBH task categories and the number of correct cases in each category.

LLM Reasoning

Large language model reasoning studies how models solve multi-step problems, verify intermediate steps, and use computation or tools reliably.

See all 915 papers

MixRea: Benchmarking Explicit-Implicit Reasoning in Large Language Models
May 19, 2026Yuanqing Cai, Ziyi Huang, Minhao Liu +3LLM Evaluation LLM Prompting
Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China
Latent Thought Flow: Efficient Latent Reasoning in Large Language Models
Jun 15, 2026Xiandong Zou, Jing Huang, Jianshu Li +1RL for Language Model Reasoning Efficient Language Model Reasoning
Singapore Management University · Ant Group
Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains
May 25, 2026Hui Xie, Jie Liu, Ziyue Qiao +1Efficient Language Model Reasoning LLM Inference Acceleration
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China

May 19, 2026J/K move · Enter open · S save

MixRea: Benchmarking Explicit-Implicit Reasoning in Large Language Models

Yuanqing Cai, Ziyi Huang, Minhao Liu, Lixin Duan+2

Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China

LLM Evaluation LLM Prompting Causal Reasoning in Language Models Implicit Reasoning in Language Models Causal Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly integrated into high-stakes decision-making. Inspired by the theory of inattentional blindness in human cognition, we investigate whether LLMs, trained on human-preferred corpora that embed attentional biases, exhibit a similar limitation: failing to attend to subtle yet important contextual cues under explicit task instructions. To evaluate this, we introduce the task of explicit-implicit reasoning and present MixRea, a benchmark of 2,246 multiple-choice questions across 9 reasoning types with varying distributions of explicit and implicit information. Evaluation of 21 advanced LLMs shows that even the best-performing reasoning model (Gemini 2.5 Pro) achieves only 42.8% consistency, revealing widespread inattentional blindness. To mitigate this, we propose Potential Relation Completion Prompting (PRCP), a prompting method that improves reasoning by recovering overlooked causal relations. Further analysis shows that this limitation persists across diverse multi-source reasoning tasks, highlighting the need for more cognitively aligned models.

Abstract

LLM Reasoning

Explore similar work

MixRea: Benchmarking Explicit-Implicit Reasoning in Large Language Models

Latent Thought Flow: Efficient Latent Reasoning in Large Language Models

Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains