In Dialogue with Intelligence: Toward Insightful Co-Augmentation

Eleni Vasilaki

cs.HCMay 28, 2025

In Dialogue with Intelligence: Toward Insightful Co-Augmentation

Authors: Eleni Vasilaki

Abstract

Dialogue with a large language model can lead a person to insight: a sudden change in how they understand a problem. This perspective asks how model activity relates to insight as a dialogue unfolds. I propose that part of the intelligence expressed in dialogue arises from two interacting recurrences: each generated token becomes context for the next, and each response returns through the person, whose interpretation and new observations reshape what the model receives. Within this loop, the model's contribution shifts between modes, from echoing familiar formulations to offering a framing that opens a new direction for the person to develop. These modes may correspond to distinguishable patterns of model activity. A memory "spine" that selects which context is carried forward could elicit productive patterns again while the ideas themselves change. Public records of human-model dialogue, activity recorded from open-weight models and tools from computational neuroscience make these proposals testable.

Figures & tables

Panel of Figure 1 — Figure 1: Context selection and model activity in insightful dialogue. A model response can help a person connect further observations; the resulting interpretation returns through the next prompt. The proposed role of the “spine” is to carry forward context that helps elicit activity supporting further connections. The two network diagrams illustrate partly overlapping patterns of activity in the same model across successive exchanges, with learned weights held fixed. In the later network, teal paths indicate shared activity and gold paths newly recruited activity. The patterns are schematic and may involve distributed representations across shared components. Dashed arrows mark the proposed relation between model activity and the model’s contributions.

Explore similar work

They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It
Jul 3, 2026Alex KwonInterpretability Readout
Independent Researcher
Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision
Jun 30, 2026Zifan Carl Guo, Laura Ruis, Jacob Andreas +1Introspection Adapters Explainable AI Methods
MIT EECS
Think-Aloud Reshapes Automated Cognitive Model Discovery Beyond Behavior
May 6, 2026Hanbo Xie, Akshay K. Jagadish, Lan Pan +1Cognitive Science Large Reasoning Models
School of Psychology, Georgia Institute of Technology · Princeton AI Lab, Princeton University · Center of Excellence for Computational Cognition, Georgia Institute of Techonology

Jul 3, 2026, cs.CLJ/K move · Enter open · S save

They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It

Alex Kwon

Independent Researcher

Interpretability Readout

Open paper AskPDF arXiv

When a person shares something with a language model, the model often answers the surface of the message rather than what the sender was doing by sending it: share a finished project and it critiques the code; share a raw late-night line and it runs a wellness check. We treat the sender's communicative intent, the Gricean what-was-meant, as a first-class interpretability object, and show the failure is one of readout on top of a robust representation. A linear probe decodes the sender's intent, whether they want a thing recognized or evaluated, from a model's default-pass hidden states, cleanly and surface-independently, across six models and four families and in the base checkpoints. The representation generalizes further, to intent that is only pragmatically inferred, and to a second, lexically clean intent (support versus help). The behavioral half of the story, and every causal test, is established on the recognize/evaluate contrast, where what varies is whether the default output acts on the intent. The readout lags the representation in depth within a model (the intent is decodable several layers before it drives the output); across models, which ones act on it by default is model-specific, an observed stratification (three of six show the failure) that we do not read as a scaling law. Where the gap is open, a direction closely tied to the representation, the discriminative direction at a searched-for layer, is a causal handle: steering it recovers the intended behavior, as well as an explicit instruction does and with no prompt at all. This direction is near-orthogonal to the feedback-offering axis, so it routes a represented intent rather than a generic feedback knob, though at the recovery dose the routed intent can override an explicit request. We support each link with controls against obvious deflations and report the nulls as plainly as the confirmations.

Abstract

Explore similar work

They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It

Introspective Coupling: Self-Explanation Training Tracks Behavioral Change Despite Fixed Supervision

Think-Aloud Reshapes Automated Cognitive Model Discovery Beyond Behavior