cs.LGMar 10, 2026

Correction of Transformer-Based Models with Smoothing Pseudo-Projector

Authors: Vitaly Bulgakov

Organizations: Profiteya LLC, Boston, MA, USA · Mass General Brigham, Boston, MA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The pseudo-projector is a lightweight modification that can be integrated into existing language models and other neural networks without altering their core architecture. It can be viewed as a hidden-representation corrector that reduces sensitivity to noise by suppressing directions induced by label-irrelevant input content. The design is inspired by the multigrid (MG) paradigm, originally developed to accelerate the convergence of iterative solvers for partial differential equations and boundary value problems, and later extended to more general linear systems through algebraic multigrid methods. We refer to the method as a pseudo-projector because its linear prototype corresponds to a strictly idempotent orthogonal projector, whereas the practical formulation employs learnable restriction and prolongation operators and therefore does not, in general, satisfy the properties of an exact orthogonal projection. We evaluate the proposed approach on transformer-based text classification tasks, as well as controlled synthetic benchmarks, demonstrating its effectiveness in improving training dynamics and robustness. Experimental results, together with supporting theoretical heuristics, indicate consistent improvements in training behavior across a range of settings, with no adverse effects observed otherwise. Our next step will be to extend this approach to language models.

Figures & tables

Figure 1: Fig 1. “Wiggly curve” model structure
Figure 2: Figure 2 Input parameters in test 1 including train and test sizes, batch size, hidden layers size, restriction (coarse) dimension, number of hidden layers, number of epochs and learning rate.
Figure 3: Figure 3 compares the learned decision boundaries of the base model with and without the projector in test1.
Figure 4: Figure 4 Validation metrics and losses with training history in test1.
Figure 5: Figure 5 compares the learned decision boundaries of the base model with and without the projector in test2 with 800 and 200 training and test samples accordingly after 15 epochs in Test 2, where solid blue line shows learned boundaries.
Figure 6: Figure 6 compares the learned decision boundaries of the base model with and without the projector in test2 with 8000 and 2000 training and test samples accordingly after 15 epochs in Test 3.
Figure 7: Figure 7 compares the learned decision boundaries of the base model with and without the projector in test2 with 8000 and 2000 training and test samples accordingly after 40 epochs in Test3.
Figure 8: Figure 8 Validation metrics and losses with training history in Test3.
Figure 9: Figure 9. Test 3 with 40 epochs and 2 projection steps. Additional smoothing by the 2nd projection steps made the learned boundary and original decision boundary almost coincide.
Figure 10: Figure 10. Simplified model workflow
Figure 11: Figure 11a. QQP dataset, Case 1 training / evaluation history
Figure 12: Figure 11b. QQP dataset, Case 1. Gradient norms.
Figure 13: Figure 12. QQP dataset, Case 1. When averaged across the full training trajectory, the Projector model consistently outperforms the Plain baseline in accuracy, precision, recall, and F1-score, indicating superior expected performance despite higher variance.
Figure 14: Figure 13a. QQP dataset, Case 2 training / evaluation history
Figure 15: Figure 13b. QQP dataset, Case 2. Gradient norms.
Figure 16: Figure 14. QQP dataset, Case 2. When averaged across the full training trajectory, the Projector model consistently outperforms the Plain baseline in accuracy, precision, recall, and F1-score, indicating superior expected performance despite higher variance.
Figure 17: Figure 15. Epoch-wise differences in Accuracy and F1 score between Proj and Plain models.
Figure 18: Figure 16. QQP dataset, Case 3 training / evaluation history
Figure 19: Figure 17. QQP dataset, Case 3. Averaged validation metrics across the full training trajectory.
Figure 20: Figure 18. QQP dataset, Case 3. “Best Epoch” in terms of Accuracy and F1-score in training competing models
Figure 21: Figure 19. SNLI dataset, training / evaluation metrics history
Figure 22: Figure 20. The much higher (this is a log graph) early gradient norms in the Proj model are consistent with a multigrid-like coarse correction phase, where global error components are addressed first, followed by fine-scale refinement as training converges.
Figure 23: Figure 21 shows that the projector-corrected model reaches its maximum performance metrics at an early stage, whereas the plain model never achieves acceptable recall.
Figure 24: Figure 22. Behavior of the weights α0\alpha_{0} , α1\alpha_{1} , α2\alpha_{2} over training epochs.
Figure 25: Figure 23. MIMIC-IV discharge summaries dataset, training / evaluation metrics history
More in this topic

Noise Robustness

Noise robustness studies how to preserve reliable predictions when data, labels, signals, or computing hardware are corrupted.

See all 93 papers

Explore similar work

CardsList

  1. Prototype-Rectified Iterative Self-supervised Manifold Denoising under Severe Acoustic Shift

    Aug 15, 2026Ashish Anand Shukla, Rini Smita Thakur, Aryan Das +1Test-Time AdaptationAudio-Language Models

    Indian Institute of Science Education and Research Bhopal, India · Vellore Institute of Technology Bhopal, India

  2. Projector Is All You Train

    Aug 20, 2026Nyx Iskandar, Saathvik Selvan, Slater VictoroffMultimodal Large Language ModelsVLM Adaptation

  3. A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning

    Oct 3, 2026Yuyang Zhang, Xiaoyin Chen, Chunlin Ren +1LLM Fine-TuningCatastrophic Forgetting

    McGill University Mila – Quebec AI Institute · Université de Montréal · Nanyang Technological University