A fundamental bottleneck for translating thoughts to speech is the high physical cost of motor input forcing a strict tradeoff between communication speed and expressivity. Generative AI offers a promising solution by expanding highly abbreviated text input into fluent sentences, significantly reducing the motor burden on the user. However, sparse inputs are inherently ambiguous; without sufficient context, AI models frequently misinterpret these abbreviations and generate outputs that fail to align with the user's true cognitive intent. We propose Cognitive Steering, a framework that can be leveraged to reconstruct user intent by grounding Large Language Models (LLMs) in multimodal context and applying AI control methods. We demonstrate that aligning frontier models with selected environmental referents, emotional states, and communication motives on simulated abbreviation benchmarks achieves up to ∼88% top-5 expansion accuracy, doubling no-context baselines. We also reveal that different types of context require distinct AI control techniques: while physical objects in a scene are best provided through explicit text prompts, abstract factors like emotion may be more effectively managed by directly altering the AI's internal activations. These findings suggest a hybrid architecture for future interfaces that leverage Cognitive Steering to help users communicate both quickly and accurately, recruiting multiple AI control methods simultaneously to perfectly capture their intended communication goals.
Figures & tables
Figure 1. (A) The AAC Pareto Frontier. Traditional AAC devices force a tradeoff ( Hoag et al., 2004 ) between expressivity (Communication Expressivity, y-axis) and efficiency (Communication Efficiency, x-axis). An example of Mechanism 1 is letter-by-letter typing, which captures nuance but is slow; and an example of Mechanism 2 is preset phrases, which are fast but potentially generic. We propose that Cognitive Alignment (CA)—achieved by grounding LLM generation on human intentionality—allows the frontier to be broken as illustrated. By successfully expanding compressed inputs (abbreviations) into the intended utterances, users will better achieve high expressivity without sacrificing efficiency. (B) Keyboard expansion and typing based on eye-gaze control. A combination of bi-gram and tri-gram allows the selection of letters via a keyboard expansion through directional gestures performed by eye movement. Additional high-frequency common words can be placed for easy access on the keyboard for positional gaze detection. (C) Contextual Steering. The CCI achieves CA by injecting multimodal context (visual ego-centric view, saliency, audio) and pragmatic elements (referents, motive of communication, emotion) into the LLM inference process. A test-time intervention mechanism (e.g. prompting or steering) uses these signals to steer the LLM, ensuring the abbreviation “A G U O T” is expanded into the contextually relevant utterances that steered closer to the intended phrase like (“A GOOD USE OF TIME") or (“ALWAYS GET UP ON TIME").
Figure 2. Frontier LLMs achieve up to ∼88% top-5 expansion accuracy when aligned to pragmatic context. (A) Image-Chat dataset. (Top) Illustration of the 4 layers. L0 includes no context information and simply asks the LLM to expand the abbreviated sentence. L1 includes the emotion of Speaker A (first dialogue line) and L2 further includes the emotion of Speaker B (second dialogue line). L3 includes referents and L4 the full scene description. (Bottom) Performance for Speaker A (first row) and Speaker B (second row) under different context layers (L0–L4), and in the emotion ablated condition L4∖L2 . The y-axis represents Accuracy, and bars denote Top-1 and Top-5 performance. (B) Generated dataset. (Top) L0 is as in panel A. L1 includes the Motive of Communication (MOC), and L2 includes the speaker emotion and explicitness level. L3 includes target and distractor referents, and L4 the scene description. (Bottom) Performance across context layers and in the emotion ablated condition L4∖L2 . The y-axis represents Accuracy, and bars denote Top-1 and Top-5 performance. Statistical significance was assessed using paired Wilcoxon signed-rank tests (one-sided) with Holm–Bonferroni correction for multiple comparisons. * p < .05, ** p < .01, *** p < .001. Panel A shows the performance on the Image-Chat dataset across different context layers (L0 to L4) and an ablated condition, with bar charts for Speaker A and Speaker B. Panel B shows performance on the generated dataset across the same context layers and ablation condition. Bar charts display accuracy metrics with error bars.
Figure 3. Cognitive Steering - Prompting vs. test-time steering for alternative context layers. (A) Structured prompting is used to provide pragmatic context features (e.g., MOC, emotions, explicitness, referents) to the model at inference time. (B) Steering procedure: a context-direction vector is added to the residual stream of a chosen layer with varying strength (Steering 1–2), enabling alignment without modifying the prompt. Steering 1 steers layers 15, 20, and 25; Steering 2 steers layers 30, 35, and 38 ( OLMo-2-13B-Instruct has 40 layers ( OLMo et al., 2025 ) ). (C) Top-5 LLM-judged similarity for alternative steering approaches (bars) on the generated dataset benchmark, for L3 (MOC, emotions, explicitness) and L4 (+Referents) context layer. The y-axis represents Top-5 LLM-judged similarity. Statistical significance was assessed using paired Wilcoxon signed-rank tests (one-sided) with Holm–Bonferroni correction for multiple comparisons. * p < .05, ** p < .01, *** p < .001.
The thinking-while-speaking paradigm aims to make AI communication more human. A key challenge is maintaining fluent speech while performing deep reasoning. Our method, InterRS, tackles this by inserting reasoning steps only during natural speech generation. This requires high-quality data where reasoning and speech are precisely aligned, and the length ratio are under controlled. We introduce a novel pipeline to generate such seamlessly interleaved audio data. To train our model, we combine interleaved SFT with refined data and reinforcement learning with two new rewards: a TA-Balance Reward to manage timing and thinking-answer ratio, and a Linguistic Quality Reward to refine expression. Experiments show our approach achieves 13% better performance on mathmatical and logic benchmarks while generating instant response like a spoken-language instruct model which outputs fast CoT response. Furthermore, our method generates more natural and fluent answers than prior methods.
Large language models can be steered at inference time through prompting or activation interventions, but activation steering methods often underperform compared to prompt-based approaches. We propose a framework that formulates prompt steering as a form of activation steering and investigates whether distilling successful prompt steering behavior into simpler, interpretable models can close this gap. Our analysis reveals that popular activation steering methods are not faithful to the mechanics of prompt steering, which applies strong interventions on some tokens while barely affecting others. Based on these insights, we introduce Prompt Steering Replacement (PSR) models that estimate token-specific steering coefficients from the activations themselves and are trained to imitate prompt-based interventions. Experiments on three steering benchmarks across multiple language models show that PSR models outperform existing activation steering methods, especially when controlling for high-coherence completions, and also compare favorably to prompting on AxBench and persona steering.
Prompting guides language model behavior through the initial context, whereas activation steering often intervenes throughout generation. A natural question is whether steering can produce effects on subsequent computation similar to those of prompting. Under fixed-state attention assumptions, we establish sufficient conditions for single- and multi-token steering to match prompt-induced attention-head outputs, and characterize how changes in input representations affect this match and its approximation error. This attention-level connection leads us to ask whether, at the behavioral level, steering can also guide subsequent generation through a brief initial intervention. We study Prefix Steering, which applies existing steering directions and operators over a short span starting at the final prompt token, with no further direct intervention afterward. We examine how intervention duration and strength jointly shape the control-capability trade-off. Across four models and five tasks, intervention over a short span, even a single token, often retains much of full steering's behavioral control while better preserving general capabilities, offering a trade-off competitive with, and in some settings better than, prompting and alternative steering-strength policies. Prefix Steering also remains effective on final-answer formatting tasks after reasoning, suggesting that a brief initial intervention can influence behavior expressed well after steering ends. These findings challenge the common practice of steering every generated token and motivate a more dynamical view of activation steering, in which a brief intervention can alter the trajectory of subsequent generation without continued intervention.