Detection Transformer

Also known as DETR

Detection Transformer studies object detectors that use transformer encoders and decoders to turn image features into a set of object classes and locations. Object queries represent candidate objects, while one-to-one assignment—often Hungarian matching—trains predictions without conventional anchors or non-maximum suppression. Researchers improve these models with deformable attention, stronger multi-scale features, better query selection and matching, and adaptations for tasks such as small-object, oriented, 3D, and open-vocabulary detection. A central challenge is balancing accuracy and generalization against training and inference cost, especially in real-time or high-resolution settings.

Latest papers

92 papers6 papers in the last four weeks, up 50% on the four weeks before. 0.0% of all new papers.

Understanding Transformer-Based Vision Models via Modular Feature Inversion
Dec 9, 2024Jan Rathjens, Shirin Reyhanian, David Kappel +1Transformer Interpretability Vision Transformer
Institute for Neural Computation Faculty of Computer Science Ruhr University Bochum · Center for Cognitive Interaction Technology Faculty of Technology Bielefeld University
OrientedFormer: An End-to-End Transformer-Based Oriented Object Detector in Remote Sensing Images
Sep 29, 2024Zeyu Ding, Yong Zhou, Jiaqi Zhao +4Remote Sensing Image Understanding Detection Transformer
School of Computer Science and Technology, China University of Mining and Technology, and with Mine Digitization Engineering Research Center of the Ministry of Education, and also with Innovation Research Center of Disaster Intelligent Prevention and Emergency Rescue, China University of Mining and Technology, Xuzhou 221116, China · School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON K1N 6N5, Canada

Dec 9, 2024J/K move · Enter open · S save

Understanding Transformer-Based Vision Models via Modular Feature Inversion

Jan Rathjens, Shirin Reyhanian, David Kappel, Laurenz Wiskott

Institute for Neural Computation Faculty of Computer Science Ruhr University Bochum · Center for Cognitive Interaction Technology Faculty of Technology Bielefeld University

Transformer Interpretability Vision Transformer Detection Transformer

Open paper AskPDF arXiv

Understanding the internal mechanisms of deep neural networks remains a central challenge in machine learning. In computer vision, one promising yet only preliminarily explored approach is feature inversion via inverse networks, which reconstructs images from intermediate representations using trained inverse networks. In this study, we revisit feature inversion via inverse networks, introducing a novel, modular variant that enables a computationally more efficient application of the technique while, in some cases, producing semantically more coherent image reconstructions. We apply our method to large-scale transformer-based vision models, specifically Detection Transformer, Vision Transformer, Swin Transformer, and Data-Efficient Image Transformer, analyzing the resulting reconstructions across network depth. Our main analysis compares Detection Transformer and Vision Transformer, which exhibit the most informative differences among the evaluated architectures. At the same time, results for Swin Transformer and Data-Efficient Image Transformer support the broader applicability of our framework. Our findings reveal gradual representational changes across transformer layers as a shared characteristic of Detection Transformer and Vision Transformer, as well as systematic differences in how they preserve contextual shape and fine-grained image details, and in their robustness to color perturbations. These findings deepen understanding of transformer-based vision models and demonstrate the utility of modular feature inversion as an interpretability tool.

Latest papers

Understanding Transformer-Based Vision Models via Modular Feature Inversion

OrientedFormer: An End-to-End Transformer-Based Oriented Object Detector in Remote Sensing Images