The Transformer Paradigm: Dynamic Processing and a New Principle of Inference
Organizations: ALOPHIS – Applied Logic Philosophy and History of Science, University of Cagliari · Dipartimento di Pedagogia, Psicologia, Filosofia, University of Cagliari
Abstract
We argue that the Transformer revolution is not adequately understood as a merely quantitative expansion of deep learning, but as the emergence of a new paradigm based on a different principle of inference. We interpret the Transformer as a system in which token representations are transformed by parameterized transformations, or transforming concepts. During inference, some of these transformations are dynamically constructed from the current input: previously acquired transformations are first adapted and then applied. At the mechanical-structural level, this dynamic processing is realized through output-weight interconnections, whereby outputs of some networks determine the weights of others. The resulting mode of processing, which we call Sequence-level Interactive Dynamic Parallel Processing (SIDPP), provides a framework for explaining strong prompt sensitivity and opens research directions concerning interpretability, control, alternative AI architectures, neuromorphic systems, and possible biological realizations of dynamic processing.
Transformer Inference
Transformer inference studies how trained Transformer models produce outputs efficiently, accurately, and securely across hardware and context lengths.
See all 63 papers