A Conditional Mutual Information Analysis of Response Variety in Language Models
Organizations: Department of Chemistry, Michigan State University, East Lansing, Michigan 48824-1322, USA
Abstract
Large language models provide a variety of responses at inference (generation) that conventionally depend on a temperature parameter. Using a conditional mutual information approach, a quantitative, entropy based analysis of response variety is given. Three limiting behaviors are defined: arbiter, independent (conditionally) and neutral (deterministic) that span no to maximal mutual information. We give examples of each behavior and suggest their application to: temperature analysis, language analysis, prompt compression, and knowledge graphs.
Mutual Information
Mutual information measures how much one variable reveals about another, supporting analysis of dependence, prediction, and learned representations.
See all 28 papers