Distributional Perspective

A distributional perspective in machine learning shifts focus from single predictions to the entire probability distribution of possible outcomes. Current research emphasizes understanding and manipulating these distributions to improve model performance, particularly in areas like large language models (LLMs) where extending context windows and enhancing robustness against adversarial attacks are key challenges. This approach is proving valuable for tasks ranging from improving the accuracy and reliability of LLMs to developing more effective reinforcement learning algorithms and even informing legal frameworks for dealing with the inherent non-determinism of machine learning systems. The distributional perspective offers a powerful lens for analyzing and improving the behavior and trustworthiness of machine learning models.

Papers

October 2, 2024

Extending Context Window of Large Language Models from a Distributional Perspective
Yingsheng Wu. Yuxuan Gu, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin
Large Language Model Long Sequence LLM Benchmark Context Window Distributional Perspective

May 29, 2024

Rejection via Learning Density Ratios
Alexander Soen, Hisham Husain, Philip Schulz, Vu Nguyen
Loss Function Distributional Assumption Risky Prompt Rejection Divergence Regularization Density Ratio Model Degradation Distributional Perspective

October 23, 2023

Meaning Representations from Trajectories in Autoregressive Models
Tian Yu Liu, Matthew Trager, Alessandro Achille, Pramuditha Perera, Luca Zancato, Stefano Soatto
Autoregressive Model Handwritten Trajectory Autoregressive Language Model Modern Language Model Distributional Learning Meaning Representation Distributional Perspective

October 9, 2022

Towards Understanding and Boosting Adversarial Transferability from a Distribution Perspective
Yao Zhu, Yuefeng Chen, Xiaodan Li, Kejiang Chen, Yuan He, Xiang Tian, Bolun Zheng, Yaowu Chen, Qingming Huang
Adversarial Attack Adversarial Example Human Understanding Transferable Adversarial Attack Distributional Perspective

June 23, 2022

Non-Determinism and the Lawlessness of Machine Learning Code
A. Feder Cooper, Jonathan Frankle, Christopher De Sa
Machine Learning Inherent Randomness Distributional Approach Distributional Perspective

February 6, 2022

Exploration with Multi-Sample Target Values for Distributional Reinforcement Learning
Michael Teng, Michiel van de Panne, Frank Wood
Environment Exploration Distributional Reinforcement Learning Multi Target Quantile Temporal Difference Distributional Perspective