TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference

Jaber Jaber; Osama Jaber

cs.LGMar 22, 2026

TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inference

Authors: Jaber Jaber, Osama Jaber

Organizations: RightNow AI

Abstract

Large language models run every token through every layer, regardless of difficulty. We present TIDE, a post-training system that attaches tiny learned routers at periodic checkpoint layers and, at inference time, selects the earliest layer whose hidden state has converged for each token. TIDE requires no model retraining, works with any HuggingFace causal LM, auto-detects GPU architecture, and supports float32, float16, and bfloat16 through fused CUDA kernels. On an NVIDIA A100 with DeepSeek R1 Distill 8B, TIDE achieves 100% prefill exit rate (5% of tokens exit at layer 11, the remaining at layer 31), reduces prefill latency by 7.2%, and increases single-batch throughput by 6.6%. During autoregressive decoding, 98-99% of tokens exit early while the model correctly solves a multi-step math problem with 95 unique output tokens. On Qwen3 8B (36 layers), throughput improves by 8.1% at batch size 8. Calibration on 2,000 WikiText samples takes under 3 minutes and produces a ~4 MB router checkpoint. The system comprises 1,308 lines of Python and 1,081 lines of CUDA/C++ with 74 passing tests. Code: https://github.com/RightNow-AI/TIDE

Figures & tables

System	Decoder	Post-training	Per-token	CUDA	Universal
DeeBERT ( Xin et al., 2020 )	✗	✓	✓	✗	✗
CALM ( Schuster et al., 2022 )	Enc-Dec	✗	✓	✗	✗
LayerSkip ( Elhoushi et al., 2024 )	✓	✗	✓	✗	✗
EE-LLM ( Chen et al., 2023b )	✓	✗	✓	✗	✗
SkipDecode ( Del Corro et al., 2023 )	✓	✓	✗	✗	✗
MoD ( Raposo et al., 2024 )	✓	✗	✓	✗	✗

Table 1: Comparison of early exit systems for transformer models.

Panel of Figure 1 — Figure 1: Tide system overview. Left : one-time calibration collects hidden states from a frozen model, computes per-token cosine similarity to the final layer, and trains a binary router MLP at each checkpoint. Right : at inference, the full forward pass runs (preserving the KV cache), then routers evaluate each checkpoint post-hoc and select the earliest converged layer for logit computation. Numbered steps show execution order.

Target Model	Hidden Dim	Bottleneck
Phi-2 / TinyLlama	2048	64
Phi-2 / TinyLlama	2048	128
Phi-3-mini	3072	128
LLaMA-8B / Mistral-7B	4096	128
LLaMA-8B / Mistral-7B	4096	256
LLaMA-13B	5120	128

Table 2: Kernel template specializations. Each entry compiles a dedicated kernel for the given hidden/bottleneck dimension pair.

Model	Layers	$\theta$	Tokens	Exit%	Distribution
DeepSeek R1 Distill 8B	32	0.85	322	100%	L11:16, L31:306
DeepSeek R1 Distill 8B	32	0.50	322	100%	L11:16, L31:306
Qwen3 8B	36	0.85	155	100%	L35:155
Qwen3 8B	36	0.50	155	100%	L11:11, L23:5, L35:139

Table 3: Prefill exit rates on 16 real text prompts (A100, bf16).

Model	Metric	Baseline	Tide	$\Delta$
DeepSeek R1 8B	Latency ( $\theta{=}0.85$ )	39.08 ms	36.94 ms	$-$ 5.5%
DeepSeek R1 8B	Latency ( $\theta{=}0.50$ )	39.08 ms	36.26 ms	$-$ 7.2%
DeepSeek R1 8B	Throughput BS=1	973 tok/s	1,037 tok/s	+ 6.6%
DeepSeek R1 8B	Throughput BS=8	8,668 tok/s	7,252 tok/s	$-$ 16.3%
Qwen3 8B	Latency ( $\theta{=}0.85$ )	46.82 ms	44.14 ms	$-$ 5.7%
Qwen3 8B	Throughput BS=1	258 tok/s	271 tok/s	+5.0%

Table 4: Prefill latency and throughput (A100, bf16, 20 runs).

\Delta

is relative to vanilla model baseline.

$\theta$	Exit Rate	Unique Tokens	Correct	Exit Layer
1.0 (off)	0%	99	✓	—
0.85	98.4%	95	✓	L31
0.70	99.2%	95	✓	L31
0.50	99.6%	95	✓	L31

Table 5: Decode exit rates and output quality on a math word problem (DeepSeek R1 Distill 8B, 256 tokens, temperature=0, A100).

Panel of Figure 2 — Figure 2: Per-token early exit in a 32-layer model. Token “the” converges at layer 8 and skips the remaining 24 layers. Token “cat” exits at layer 16. Only “sat” requires full depth. Tide evaluates the routers $\phi_{k}$ post-hoc after the full forward pass completes.

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 276 papers

River-LLM: Large Language Model Seamless Exit Based on KV Share
Apr 20, 2026Yingtao Shen, An ZouDecoder-Only Language Models KV-Cache Management
Shanghai Jiao Tong University Shanghai, China
TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
Feb 5, 2026Jiyoung Park, Hankyu Jang, Changseok Song +1Adaptive Inference LLM Inference Serving
Moreh, Inc.
TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload
May 19, 2026Zhiben Chen, Youpeng Zhao, Yang Sui +2Expert Offloading LLM Inference Scheduling
University of Central Florida · Rice University

Apr 20, 2026J/K move · Enter open · S save

River-LLM: Large Language Model Seamless Exit Based on KV Share

Yingtao Shen, An Zou

Shanghai Jiao Tong University Shanghai, China

Decoder-Only Language Models KV-Cache Management LLM Inference Acceleration Early-Exit Neural Networks

Open paper AskPDF arXiv

Large Language Models (LLMs) have demonstrated exceptional performance across diverse domains but are increasingly constrained by high inference latency. Early Exit has emerged as a promising solution to accelerate inference by dynamically bypassing redundant layers. However, in decoder-only architectures, the efficiency of Early Exit is severely bottlenecked by the KV Cache Absence problem, where skipped layers fail to provide the necessary historical states for subsequent tokens. Existing solutions, such as recomputation or masking, either introduce significant latency overhead or incur severe precision loss, failing to bridge the gap between theoretical layer reduction and practical wall-clock speedup. In this paper, we propose River-LLM, a training-free framework that enables seamless token-level Early Exit. River-LLM introduces a lightweight KV-Shared Exit River that allows the backbone's missing KV cache to be naturally generated and preserved during the exit process, eliminating the need for costly recovery operations. Furthermore, we utilize state transition similarity within decoder blocks to predict cumulative KV errors and guide precise exit decisions. Extensive experiments on mathematical reasoning and code generation tasks demonstrate that River-LLM achieves 1.53 to 2.16 times of practical speedup while maintaining high generation quality.

Abstract

LLM Inference

Explore similar work

River-LLM: Large Language Model Seamless Exit Based on KV Share

TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference

TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload