Prediction Head

A prediction head is a crucial component in many machine learning models, responsible for transforming internal feature representations into final predictions. Current research focuses on improving prediction head design across diverse tasks, including time series forecasting (using transformers and CNN-based architectures like PatchMixer), natural language processing (analyzing bias effects on word frequency prediction), and computer vision (exploring explainable and calibrated approaches like the Nadaraya-Watson head). These advancements aim to enhance model accuracy, efficiency, interpretability, and robustness, impacting fields ranging from document analysis to image recognition and beyond.

Papers

August 2, 2024

UNER: A Unified Prediction Head for Named Entity Recognition in Visually-rich Documents
Yi Tu, Chong Zhang, Ya Guo, Huan Chen, Jinyang Tang, Huijia Zhu, Qi Zhang
Entity Recognition Visually Rich Document Transformer Backbone Entity Relation Extraction Prediction Head Entity Recognition Performance

February 26, 2024

Generative Pretrained Hierarchical Transformer for Time Series Forecasting
Zhiding Liu, Jiqian Yang, Mingyue Cheng, Yucong Luo, Zhi Li
Time Series Forecasting Hierarchical Transformer Auto Regressive Prediction Head

October 1, 2023

PatchMixer: A Patch-Mixing Architecture for Long-Term Time Series Forecasting
Zeying Gong, Yujin Tang, Junwei Liang
Neural Network Architecture Design Long Term Time Series Forecasting Forecasting Benchmark Prediction Head

May 29, 2023

Transformer Language Models Handle Word Frequency in Prediction Head
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro Inui
Transformer Based Language Model Transformer Language Model Long Tailed Learning Autoregressive Text Generation Prediction Head

March 13, 2023

Hierarchical Relationships: A New Perspective to Enhance Scene Graph Generation
Bowen Jiang, Camillo J. Taylor
Scene Graph Generation New Perspective Hierarchical Structure Visual Genome Hierarchical Information Hierarchical Relationship Prediction Head

December 7, 2022

A Flexible Nadaraya-Watson Head Can Offer Explainable and Calibrated Classification
Alan Q. Wang, Mert R. Sabuncu
Inherent Interpretability Neural Network Architecture Human Head Classification Head Calibrated Learning Nadaraya Watson Prediction Head

November 30, 2022

Boosted Dynamic Neural Networks
Haichao Yu, Haoxiang Li, Gang Hua, Gao Huang, Humphrey Shi
Boosting Algorithm Dynamic Neural Network Prediction Head

September 26, 2022

End-to-end Multilingual Coreference Resolution with Mention Head Prediction
Ondřej Pražák, Miloslav Konopík
Coreference Resolution Question Based Dependency Multilingual Training Prediction Head

May 12, 2022

The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning
Zixin Wen, Yuanzhi Li
Neural Network Contrastive Loss Meaningful Representation Functional Mechanism Non Contrastive Self Supervised Learning Prediction Head

February 7, 2022

HeadPosr: End-to-end Trainable Head Pose Estimation using Transformer Encoders
Naina Dhingra
Human Head Encoder Side Transformer Encoder Transformer Encoders Head Pose Estimation Position Embeddings Prediction Head