cs.CLAug 28, 2026
SaveA Formal Limitation on Learning Human Language From Textual Corpora
Organizations: Universitat Pompeu Fabra · ETH Zürich
Abstract
Can a listener recover what a speaker means from the form of an utterance alone? We answer this question information-theoretically, and for a listener given by any featurizer of text, including the hidden states of contemporary large language models. Modeling language use as a joint distribution over meanings, contexts, and utterances, we derive upper bounds on the probability that a decoder recovers a speaker's intended meaning from a representation of the utterance. The bounds are governed by the uncertainty that form leaves about meaning, which splits into an irreducible part and a part that only (extralinguistic) context, but never the utterance alone, can resolve. Because these quantities are intrinsic to language, no representation, however much text or supervision produced it, can surpass them. The bounds apply, moreover, to meaning spaces that are discrete or continuous. We provide empirical evidence in support of the theory through experiments on artificial languages, Mandarin zero-pronoun resolution, and color reference.
Figures & tables
Figure 1: Communication model. During communication, a context {\color[rgb]{0.8398,0.1523,0.1563}C} influences a speaker’s intended meaning, given by random variable {\color[rgb]{0.1211,0.4648,0.707}M} , which the speaker encodes into an utterance {\color[rgb]{0.1719,0.6289,0.1719}U} given that same context. Under the same context {\color[rgb]{0.8398,0.1523,0.1563}C} , the listener decodes the utterance {\color[rgb]{0.1719,0.6289,0.1719}U} into their inferred meaning \widehat{{\color[rgb]{0.1211,0.4648,0.707}M}} . Communication is considered successful if {\color[rgb]{0.1211,0.4648,0.707}M}\approx\widehat{{\color[rgb]{0.1211,0.4648,0.707}M}} .
Figure 2: Experiments. A: (Left) Artificial languages over a categorical meaning space. In a shared categorical context {\color[rgb]{0.8398,0.1523,0.1563}C}= , the speaker (left) intends meaning {\color[rgb]{0.1211,0.4648,0.707}M}=\blacksquare and produces utterance {\color[rgb]{0.1719,0.6289,0.1719}U}= “a”. The listener decodes {\color[rgb]{0.1719,0.6289,0.1719}U} in context back into the original meaning. (Right) For a family of six artificial languages (dots on the curves) that vary \mathrm{I}({\color[rgb]{0.1211,0.4648,0.707}M};{\color[rgb]{0.1719,0.6289,0.1719}U}) (x-axis), the theoretical upper bound (black) exceeds the best empirical test accuracy across runs (blue), validating Theorem 1 . B: (Left) Artificial languages over continuous meanings in the -recovery setting. In shared categorical context {\color[rgb]{0.8398,0.1523,0.1563}C}= , the speaker intends a 2D coordinate {\color[rgb]{0.1211,0.4648,0.707}M}\sim\textrm{Unif}[-1,1]^{2} and produces the utterance {\color[rgb]{0.1719,0.6289,0.1719}U}= “b”. The listener approximates {\color[rgb]{0.1211,0.4648,0.707}M} given utterance in context. (Right) Theorem 2 holds across eight artificial languages varying over \mathrm{I}({\color[rgb]{0.2031,0.2734,0.9219}J}_{\epsilon};{\color[rgb]{0.1719,0.6289,0.1719}U}) , seen by the theoretical bound exceeding the empirical test -accuracy ( ). C: (Left) Zero-pronoun resolution in Mandarin Chinese. In a context where the speaker and listener just had dinner together, the speaker asks the listener whether she has chewing gum, dropping the pronoun (you) from the utterance. The listener infers the true pronoun in context. (Right) For zero-pronoun resolution from LLM and VLM activations, Theorem 1 holds: the predicted accuracy ceiling ( , dashed line) exceeds the best empirical test accuracies across runs (blue bars). D: (Left) Color naming in context. Given a shared context {\color[rgb]{0.8398,0.1523,0.1563}C} of three color chips, the speaker intends the target chip {\color[rgb]{0.1211,0.4648,0.707}M}= and says {\color[rgb]{0.1719,0.6289,0.1719}U}= “Darkish red”, which the listener decodes in context to click on the target. (Right) Theorem 2 ’s ceiling ( , dashed line) for -accurate recovery ( ) exceeds all models’ best-case empirical performances over runs (orange bars).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Mandarin pronouns distribution preprocessed from Wang et al. (2018) . Percentage annotations (black) are each pronoun’s percentage of the total. The orange percentage annotations are the percent of each pronoun that is dropped (corresponding to the orange segment in the bars). For example, 我 (‘I’) is 34% of all true pronouns; it is dropped 15.1% of the time.
Figure 4: Artificial languages over continuous meanings; other values of . We show that the Theorem 2 bound (black line) holds for various settings of .
Figure 5: Theorem 2 tightness, color-naming. As the parameter (x-axis), which controls the granularity of the quantization, increases, the tightness of the bound in Theorem 2 (y-axis) improves. The number of bins |{\color[rgb]{0.2031,0.2734,0.9219}J}_{\epsilon}| in the quantization implied by is shown at the top x-axis; at the right extreme case , the -accurate regression task reduces to binary classification ( |{\color[rgb]{0.2031,0.2734,0.9219}J}_{\epsilon}|=2 ). At the left extreme, the task is |{\color[rgb]{0.2031,0.2734,0.9219}J}_{\epsilon}|=3253 -way classification. We observe that Theorem 2 becomes “useful” in an intermediate regime, .
Figure 6: Color naming, Theorem 2 for all values of . Analogue to Figure 2 (left) for each value of . Bounds hold for all values of and typically tighten with larger ; for reference, the number of bins |{\color[rgb]{0.2031,0.2734,0.9219}J}_{\epsilon}| in the quantized meaning variable {\color[rgb]{0.2031,0.2734,0.9219}J}_{\epsilon} is also shown.