cs.CLAug 4, 2026

LoopMTP: A looped transformer guided by latent multi-token prediction

Authors: Behzad ShomaliMarkus FreyDavid BerghausJoachim KoehlerMehdi Ali

Organizations: Lamarr Institute · University of Bonn · Fraunhofer IAIS

Abstract

Looped transformers have emerged as a parameter-efficient alternative to scaling depth for strong reasoning. By reusing one stack of layers across TT iterations, they attain the effective depth and reasoning capabilities of larger models at a fixed parameter count. Yet existing approaches suffer from latent overthinking and undifferentiated computation, largely because intermediate representations receive no guidance across loops. Multi-token prediction (MTP) supplies exactly the dense, forward-looking supervision the loop is missing. We propose \textsc{LoopMTP}, which links the two through a structural correspondence in latent space: a model that loops TT times can anticipate TT future tokens. \textsc{LoopMTP} realizes this by softly aligning the hidden state of loop tt with the embedding of the token tt steps ahead, while a lightweight gate preserves useful information across iterations. \textsc{LoopMTP} improves average accuracy by up to 8.1% (relative) over the non-looped baseline, with training remaining stable for up to 15 loops.

Explore similar work

CardsList