Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers

Mohan Tang; Sidi Lu

cs.LGFeb 20, 2026

Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers

Authors: Mohan Tang, Sidi Lu

Organizations: UCLA.

Abstract

Complex problems, whether in math, logic, or planning, are solved by humans through a sequence of steps where the result of one step informs the next. In this work, we adopt the perspective that the reasoning power of Transformers is fundamentally limited by a fixed maximum number of steps along any latent path of computation. To address this, we introduce Turbo Connection (TurboConn), a novel architecture that overcomes the fixed-depth constraint by routing multiple residual connections from the higher-layer hidden states of each token $t$ to the lower layers of token $t+1$ . Fine-tuning pre-trained LLMs with our method not only yields accuracy gains of 0.9% to over 10% on benchmarks like GSM8K, Parity, and multi-step arithmetic, but also demonstrates that the density of these backward connections is critical; our dense interaction significantly outperforms "sparse" alternatives that only pass a single hidden state or vector. Notably, TurboConn can be integrated into pre-trained LLMs to overcome task-specific plateaus: while a fine-tuned Qwen-3-1.7B achieves only 53.78% on Parity, adding our architectural modification enables the model to reach 100% accuracy, all without the necessity to retrain the full model from scratch or sophisticated curriculum learning. Our results provide strong empirical evidence that the depth of the computational path is a key factor in reasoning ability, also offering a new mechanism to enhance LLMs without significantly affecting generation latency.

Figures & tables

Figure 1: Modified Transformer architecture with downward connections (orange arrows) from higher to lower decoder layers.

Figure 2: Grouping strategy for downward connections. Example shown for group of 2.

Figure 3: This diagram illustrates the dense connectivity pattern utilized in our experimental setup, depicted here for a group size of 1.

Dataset (Avg. Seq. Length)	Method	# Recursions	Time/Step (seconds) (× Transformer)	Acc (%) ( $\alpha=1$ )	Acc (%) ( $\alpha=100$ )
Parity (154.69 tokens)	Baseline	1.00	1.18 (1.00×)	92.87	–
	TurboConn (Group 4)	38.81	5.75 (4.87×)	98.94	100.00
	TurboConn (Group 6)	25.94	3.07 (2.60×)	98.68	100.00
	TurboConn (Group 8)	19.84	2.66 (2.25×)	98.83	51.42
	TurboConn (Group 16)	10.00	1.61 (1.36×)	98.85	51.40
Multi-step Arithmetic (125.75 tokens)	Baseline	1.00	1.01 (1.00×)	38.16	–

Table 1: Training efficiency and performance analysis for Llama 3.2 1B across group sizes for multipliers

\alpha=1

and

\alpha=100

. We report average sequence length, average number of recursions per training iteration, and relative training time per step compared to the baseline transformer across three reasoning tasks.

Model	Dataset	Method	Acc (%)
Llama 3.2 1B	GSM8K (No CoT)	Baseline	7.20
	GSM8K (No CoT)	TurboConn	8.32
	Multi-step- arithmetic	Baseline	38.16
	Multi-step- arithmetic	TurboConn	42.66
	Parity	Baseline	92.87
	Parity	TurboConn	100.0

Table 2: Performance comparison across model sizes and datasets, comparing standard Transformers baseline against Transformers augmented with TurboConn.

Model	Dataset	Method	Acc (%)
Llama 3.2 1B	GSM8K (No CoT)	Baseline	7.20
		Soft-token	7.07
		TurboConn	8.32
	Multi-step Arithmetic	Baseline	38.16
		Soft-token	38.00
		TurboConn	42.66

Table 3: Comparison of our dense feedback method against standard Transformers baseline and the single soft-token feedback using Llama 3.2 1B.

Figure 4: Length Generalization Performance. We evaluate a Llama 3.1 8B model, trained on Parity with up to 10-digit sequences, on its ability to generalize to longer input sequences.

Model	Method	Acc (%)	Eliminated Choices
Qwen3-1.7B	Baseline	36.10	2.516
Qwen3-1.7B	TurboConn	45.81	5.259
Qwen3-4B	Baseline	43.68	4.609
Qwen3-8B	Baseline	45.61	4.500

Table 4: Comparison of accuracy and discriminative filtering on Multi-step Arithmetic after finetuning. “Eliminated Choices” represents the mean number of digit classes with probability

P<0.001

. The result for TurboConn is reported using a group size of 4.

Task	Method	Acc (%)
NuminaMath-CoT	Baseline + CoT	30.96
NuminaMath-CoT	TurboConn + CoT	33.98
GSM8K	Baseline + CoT	36.45
GSM8K	TurboConn + CoT	36.96
AMC & AIME (Transferred)	Baseline + CoT	2.00
AMC & AIME (Transferred)	TurboConn + CoT	4.45

Table 5: Performance comparison using generated Chain-of-Thought trajectories. All results use Llama 3.2-1B; TurboConn uses a group size of 8.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Module	$\boldsymbol{d_{in}}$	$\boldsymbol{d_{out}}$
Attention Block
q_proj	$d_{hidden}$	$d_{hidden}$
k_proj	$d_{hidden}$	$d_{kv}$
v_proj	$d_{hidden}$	$d_{kv}$
o_proj	$d_{hidden}$	$d_{hidden}$
MLP Block

Appendix

Table 6: Dimensions of the layers adapted with LoRA.

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 540 papers

A transformer architecture alteration to incentivise externalised reasoning
Mar 22, 2026Elizabeth Pavlova, Mariia Koroliuk, Karthik Viswanathan +3Efficient Language Model Inference Large Reasoning Models
MARS · Geodesic Research, equal contribution · University of Amsterdam
Do Transformers Use their Depth Adaptively? Evidence from a Relational Reasoning Task
Apr 14, 2026Alicia Curth, Rachel Lawrence, Sushrut Karmalkar +1Adaptive Computation Transformer Interpretability
Microsoft Research Cambridge
Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
Apr 9, 2026Harsh Kohli, Srinivasan Parthasarathy, Huan Sun +1Position Extrapolation Implicit Reasoning in Language Models
The Ohio State University

Mar 22, 2026J/K move · Enter open · S save

A transformer architecture alteration to incentivise externalised reasoning

Elizabeth Pavlova, Mariia Koroliuk, Karthik Viswanathan, Cameron Tice+2

MARS · Geodesic Research, equal contribution · University of Amsterdam

Efficient Language Model Inference Large Reasoning Models Efficient Transformer Inference

Open paper AskPDF arXiv

We propose a new architectural change, and post-training pipeline, for making LLMs more verbose reasoners by teaching a model to truncate forward passes early. We augment an existing transformer architecture with an early-exit mechanism at intermediate layers and train the model to exit at shallower layers when the next token can be predicted without deep computation. After a calibration stage, we incentivise the model to exit as early as possible while maintaining task performance using reinforcement learning. We provide preliminary results to this effect for small reasoning models, showing that they learn to adaptively reduce computations across tokens. We predict that, applied at the right scale, our approach can minimise the amount of excess computation that reasoning models have at their disposal to perform non-myopic planning using their internal activations, reserving this only for difficult-to-predict tokens.

Abstract

Transformer

Explore similar work

A transformer architecture alteration to incentivise externalised reasoning

Do Transformers Use their Depth Adaptively? Evidence from a Relational Reasoning Task

Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers