cs.LGSep 29, 2026

Procedural Core: A Compact Recurrent Initialization for Vision Transformers

Authors: Zachary Shinnick, Christian Internò, Hemanth Saratchandran, Anton van den Hengel, Damien Teney

Organizations: Australian Institute for Machine Learning (AIML), Adelaide University · Bielefeld University · Metacognition AI · Idiap Research Institute

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Transformers are typically trained from random initialization, requiring all their capabilities to emerge from large-scale optimization. Recent work showed that a small amount of abstract procedurally generated data can help acquire generic inductive structure at low cost. However, this adds a pretraining stage that must be repeated for every target model. We propose Procedural Core, an initialization strategy that captures this generic structure into a compact set of weights that can be reused across models. We train a minimal recurrent transformer on procedural data, then expand its weights to initialize transformers of arbitrary width and depth. The resulting initialization improves performance on image classification, self-supervised visual learning (DINO), and modeling natural language (FineWeb-Edu) and code (CodeParrot). For image classification, expanding a 1M-parameter core to initialize an 85M-parameter ViT-Base improves ImageNet top-1 accuracy by 2.2 pp over standard random initialization. Our analysis identifies recurrence as essential for learning compact weights that transfer across models. In ViTs, we localize a key benefit in the suppression of high-norm tokens that produces substantial improvements in zero-shot segmentation (ImageNet-S mAP 32.3 to 42.9), object localization (VOC07 CorLoc 9.9 to 18.4), and depth estimation (NYUv2 RMSE 1.104 to 0.998). This demonstrates that transformers need not start from a blank slate, and can be initialized with generic capabilities at low cost with no domain- or task-specific data.

Figures & tables

Figure 1: We propose a generic initialization for transformers as an alternative to random weights. (Left) We first train a small auxiliary model to capture generic computations. We generate abstract training data with simple algorithms such as formal languages with nested structures (pictured as balanced parentheses). The model is recurrent in depth such that it captures regularities of the data in a minimal set of weights: the Procedural Core . (Middle) We unroll and expand the Procedural Core to initialize a target model of arbitrary width and depth. (Right) This bootstraps the subsequent standard training, improving convergence and generalization compared to a random initialization.
Figure 2: Procedural Core improves ViT-Base training on ImageNet-1k . Left: Final top-1 accuracy (mean ±\pm s.d. over three seeds). Right: Mean training trajectories over the same seeds, showing a persistent advantage over default initialization.
Figure 3: Procedural Core improves representation learning with DINO on ImageNet-1k . Left: kk -NN accuracy at selected epochs. Right: kk -NN accuracy throughout training.
Figure 4: Procedural Core improves language modeling compared to a default random initialization (lower is better; mean ±\pm std. dev. over 3 seeds).
Figure 5: Architectural choices and ablations of our method. (a) Transfer peaks with three unique auxiliary layers. (b) Removing recurrence substantially reduces transfer despite matching the parameter count. (c) The expansion in depth is effective with various number of layers in the target model. (d) The expansion in width is slightly more effective than a baseline with zero-padding.
Figure 6: Cumulative energy of singular values of different weight matrices in models trained with the Procedural Core recurrent architecture ( ) vs. a direct warm-up ( ) as in Shinnick et al. (2026) .
Figure 7: Rank truncation hinders transfer more sharply with Proc. Core weights.
Figure 8: Token dynamics across depth. Per-token hidden-state norms for default and Procedural Core initializations. High-norm outliers are suppressed from the middle blocks onwards with Procedural Core . Heatmaps are independently normalized; extended examples are in Appendix B.6 .
Figure 9: Procedural Core suppresses high-norm tokens and reduces attention to them. Statistics on 200 ImageNet-1k images. (a) Bimodality across depth. (b-c) Token-norm distributions at peak-bimodality blocks. (d) Attention mass received by high-norm tokens.
Figure 10: Transfer is concentrated in the value/output pathway. (a,b) Component-shuffling accuracy. (c,d) Shuffling V/proj. reverts to the default behavior with high-norm tokens, while shuffling Q/K preserves the Procedural Core effect.
Figure 11: Procedural Core improves representations, particularly for dense prediction tasks (segmentation, localization, and depth estimation; frozen-backbone performance). This is consistent with our finding of preserving local visual information by suppressing high-norm tokens.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

SettingValue
Batch size256
Training steps15,000
Mask ratio0.5 (close-only)
OptimizerAdamW
Learning rate2×10−32\times 10^{-3}
Weight decay0.05
Appendix
Table 1: Optimization settings used to train the auxiliary model for learning the Procedural Core.
SettingValue
ArchitectureGPT-2 Small (12L, 768D, 12H)
Context length1024
Training budget2B tokens
Effective batch size32 sequences (32,768 tokens)
OptimizerAdamW
Learning rate6×10−46\times 10^{-4} (cosine to 6×10−56\times 10^{-5} )
Appendix
Table 2: GPT-2 Small training configuration for FineWeb-Edu and CodeParrot experiments.
 Top-1 Acc. (%)
Procedural warm-up73.9±0.173.9\pm 0.1
Procedural Core (ours)74.6±0.574.6\pm 0.5
Appendix
Table 3: Comparison with procedural warm-up on CIFAR-100 . ViT-Tiny top-1 accuracy (mean ±\pm std).
InitializationTop-1 (%)
Default init.88.4±0.888.4\pm 0.8
Mimetic init.89.1±0.589.1\pm 0.5
Procedural warm-up89.4±0.389.4\pm 0.3
Procedural Core89.7±0.189.7\pm 0.1
Appendix
Table 4: Downstream transfer on CIFAR-100 . Fine-tuning accuracy after ImageNet-1K pretraining (mean ±\pm std over three seeds).
InitializationTop-1 (%)
Default initialization75.36
Procedural Core (ours)75.42
Appendix
Table 5: Linear evaluation of DINO representations on ImageNet-1K . Frozen ViT-S features evaluated with a linear classifier.
Figure 12: Layer-wise singular value spectra under recurrent vs. independent parameterization. Singular value decay for attention and MLP weight matrices across layers 1–10. In the recurrent model, the middle layers share parameters, so the same spectrum appears across layers. Each subplot contrasts this with independently parameterized layers. Recurrent layers exhibit slower spectral decay, indicating a more distributed use of singular directions.
Figure 13: Additional token-norm dynamics across depth. Per-token hidden-state norms across transformer blocks for default and Procedural Core initializations. The models begin with similar token-norm structure but increasingly diverge across depth, with the Procedural Core suppressing high-norm token outliers in later blocks.
Figure 14: Full downstream evaluation across dense and localization tasks. Frozen ViT-B backbones evaluated on segmentation ( ADE20K , ImageNet-S ), object localization ( VOC07 ), and depth estimation ( NYUv2 ). The Procedural Core improves over default initialization across the primary metrics; shuffling V/proj largely removes these gains, while transplanting V/proj+LN recovers much of them. Error bars denote 95%95\% bootstrap intervals ( NYUv2 : s.d. over three readout seeds).
Figure 15: Under the stronger DeiT-III recipe, gains concentrate on attention-based tasks. Full downstream evaluation under DeiT-III ( Touvron et al., 2022 ) . Procedural Core improves zero-shot segmentation on ImageNet-S and unsupervised object localization on VOC07 , while performance is broadly unchanged on the remaining tasks.

Explore similar work

CardsList

  1. TuringViT: Making SOTA Vision Transformers Accessible to All

    Jun 23, 2026Qiman Wu, Hanlin Chen, Lyujie Chen +19Self-Supervised Vision TransformersVision Transformer

    Foundation Model Team, Xpeng Inc.

  2. A Smaller Transformer in Your Transformer

    Sep 17, 2026Dhananjay Tomar, Marius Aasan, Andreas Kleppe +1Vision TransformerTransformer Architectures

    Institute for Cancer Genetics and Informatics Oslo University Hospital Oslo, Norway · Department of Informatics University of Oslo Oslo, Norway · SFI Visual Intelligence UiT The Arctic University of Norway Tromsø, Norway

  3. Investigating Single-Block Recurrence in Vision Transformers for Image Recognition

    May 11, 2026Michal Byra, Pawel Olszowiec, Grzegorz Stefanski +2Vision TransformerSelf-Supervised Vision Transformers

    Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland