Context Generalization

Context generalization focuses on enabling machine learning models, particularly deep neural networks, to effectively apply knowledge learned from one context to novel, unseen contexts or tasks. Current research investigates how model architectures, such as transformers and vision transformers, can be trained to generalize through mechanisms like learning compositional representations, leveraging contextual information within prompts, and employing techniques like prototypical context-aware dynamics. This research is significant because improved context generalization is crucial for building more robust and adaptable AI systems capable of handling real-world scenarios with diverse and unpredictable inputs, impacting fields ranging from natural language processing to robotics.

Papers

October 2, 2024

Bayes' Power for Explaining In-Context Learning Generalizations
Samuel Müller, Noah Hollmann, Frank Hutter
Neural Network Context Learning Real Power Bayes Rule Context Generalization

August 19, 2024

In-Context Learning with Representations: Contextual Generalization of Trained Transformers
Tong Yang, Yu Huang, Yingbin Liang, Yuejie Chi
Context Learning Meaningful Representation Task Specific Transformer Training Head Transformer Context Generalization

May 26, 2023

Im-Promptu: In-Context Composition from Image Prompts
Bhishma Dedhia, Michael Chang, Jake C. Snell, Thomas L. Griffiths, Niraj K. Jha
Compositional Generalization Visual Prompt Analogical Reasoning Visual in Context Learning Context Generalization

November 23, 2022

Prototypical context-aware dynamics generalization for high-dimensional model-based reinforcement learning
Junjie Wang, Yao Mu, Dong Li, Qichao Zhang, Dongbin Zhao, Yuzheng Zhuang, Ping Luo, Bin Wang, Jianye Hao
High Dimensional World Model Dynamic Generalization Context Generalization

October 26, 2022

SimpleDG: Simple Domain Generalization Baseline without Bells and Whistles
Zhi Lv, Bo Lin, Siyuan Liang, Lihua Wang, Mochen Yu, Yao Tang, Jiajun Liang
Domain Generalization Generalization Performance Bell and Whistle Context Generalization

March 21, 2022

ScalableViT: Rethinking the Context-oriented Generalization of Vision Transformer
Rui Yang, Hailong Ma, Jie Wu, Yansong Tang, Xuefeng Xiao, Min Zheng, Xiu Li
Vision Transformer Self Attention Vision Task Context Generalization