Language Model-Based Control

Ask about this topic

Overview

The field studies both how language models can control external systems and how controllers can regulate a model’s own behavior and outputs. Common designs use an LLM for high-level planning, intent interpretation, or code generation, paired with faster controllers such as MPC, reinforcement learning, PID, and deterministic tools; retrieval, world models, and formal safety shields help ground or check proposed actions. Researchers evaluate these systems in settings such as robot manipulation, autonomous driving, traffic signals, networks, HVAC, and software agents, using benchmarks including CARLA, SUMO, LIBERO, nuPlan, and OSWorld as well as some hardware tests. Results show useful planning and adaptation, but reliable deployment remains limited by latency, unsafe or incorrect actions, and gaps between simulation performance and robust real-world control.

Latest papers

CardsList

180 papers29 papers in the last four weeks, up 71% on the four weeks before. 0.2% of all new papers.

Mar 26, 2026

Large Language Models as Optimization Controllers: Adaptive Continuation for SIMP Topology Optimization

We present a framework in which a large language model (LLM) acts as an online adaptive controller for SIMP topology optimization, replacing conventional fixed-schedule continuation with real-time, state-conditioned parameter decisions. At every kk-th iteration, the LLM receives a structured observation−-current compliance, grayness index, stagnation counter, checkerboard measure, volume fraction, and budget consumption−-and outputs numerical values for the penalization exponent pp, projection sharpness ββ, filter radius rmin⁡r_{\min}, and move limit δδ via a Direct Numeric Control interface. A hard grayness gate prevents premature binarization, and a meta-optimization loop uses a second LLM pass to tune the agent's call frequency and gate threshold across runs. We benchmark the agent against four baselines−-fixed (no-continuation), standard three-field continuation, an expert heuristic, and a schedule-only ablation−-on three 2-D problems (cantilever, MBB beam, L-bracket) at 120 ⁣× ⁣60120\!\times\!60 resolution and two 3-D problems (cantilever, MBB beam) at 40 ⁣× ⁣20 ⁣× ⁣1040\!\times\!20\!\times\!10 resolution, all run for 300 iterations. A standardized 40-iteration sharpening tail is applied from the best valid snapshot so that compliance differences reflect only the exploration phase. The LLM agent achieves the lowest final compliance on every benchmark: −5.7%-5.7\% to −18.1%-18.1\% relative to the fixed baseline, with all solutions fully binary. The schedule-only ablation underperforms the fixed baseline on two of three problems, confirming that the LLM's real-time intervention−-not the schedule geometry−-drives the gain. Code and reproduction scripts will be released upon publication.

Shaoliang Yang, Jun Wang, Yunsheng Wang

Department of Mechanical Engineering, Santa Clara University, Santa Clara, CA 95053, USA

Topology OptimizationLarge Language Model-Guided OptimizationLanguage Model-Based Control
AskPDF
Mar 25, 2026

Toward Full Autonomous Laboratory Instrumentation Control with Large Language Models

The control of complex laboratory instrumentation often requires significant programming expertise, creating a barrier for researchers lacking computational skills. This work explores the potential of large language models (LLMs), such as ChatGPT, and LLM-based artificial intelligence (AI) agents to enable efficient programming and automation of scientific equipment. Through a case study involving the implementation of a setup that can be used as a single-pixel camera or a scanning photocurrent microscope, we demonstrate how ChatGPT can facilitate the creation of custom scripts for instrumentation control, significantly reducing the technical barrier for experimental customization. Building on this capability, we further illustrate how LLM-assisted tools can be extended into autonomous AI agents capable of independently operating laboratory instruments and iteratively refining control strategies. This approach underscores the transformative role of LLM-based tools and AI agents in democratizing laboratory automation and accelerating scientific progress.

Yong Xie, Kexin He, Andres Castellanos-Gomez

Key Laboratory of Wide Band-Gap Semiconductor Technology, School of Advanced Materials and Nanotechnology, Xidian University, Xi’an 710071, China · 2D Foundry Research Group. Instituto de Ciencia de Materiales de Madrid (ICMM-CSIC), Madrid, E-28049, Spain.

Laboratory AutomationAutonomous ExperimentationLanguage Model-Based ControlLLM Agents
AskPDF
Mar 24, 2026

AI-driven Intent-Based Networking Approach for Self-configuration of Next Generation Networks

Intent-Based Networking (IBN) aims to simplify operating heterogeneous infrastructures by translating high-level intents into enforceable policies and assuring compliance. However, dependable automation remains difficult because (i) realizing intents from ambiguous natural language into controller-ready policies is brittle and prone to conflicts and unintended side effects, and (ii) assurance is often reactive and struggles in multi-intent settings where faults create cascading symptoms and ambiguous telemetry. This paper proposes an end-to-end closed-loop IBN pipeline that uses large language models with structured validation for natural language to policy realization and conflict-aware activation, and reformulates assurance as proactive multi-intent failure prediction with root-cause disambiguation. The expected outcome is operator-trustworthy automation that provides actionable early warnings, interpretable explanations, and measurable lead time for remediation.

Md. Kamrul Hossain, Walid Aljoby

Information and Computer Science Department, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia · IRC for Intelligent Secure Systems, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia

Intent-Based NetworkingLanguage Model-Based ControlRoot Cause Analysis
AskPDF
Mar 24, 2026

Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models

Deploying safety-critical agents requires anticipating the consequences of actions before they are executed. While world models offer a paradigm for this proactive foresight, current approaches relying on visual simulation incur prohibitive latencies, often exceeding several seconds per step. In this work, we challenge the assumption that visual processing is necessary for failure prevention. We show that a trained policy's latent state, combined with its planned actions, already encodes sufficient information to anticipate action outcomes, making visual simulation redundant for failure prevention. To this end, we introduce DILLO (DIstiLLed Language-ActiOn World Model), a fast steering layer that shifts the paradigm from "simulate-then-act" to "describe-then-act." DILLO is trained via cross-modal distillation, where a privileged Vision Language Model teacher annotates offline trajectories and a latent-conditioned Large Language Model student learns to predict semantic outcomes. This creates a text-only inference path, bypassing heavy visual generation entirely, achieving a 14x speedup over baselines. Experiments on MetaWorld and LIBERO demonstrate that DILLO produces high-fidelity descriptions of the next state and is able to steer the policy, improving episode success rate by up to 15 pp and 9.3 pp on average across tasks. Code is available at github.com/MaxPappa/DILLO.

Massimiliano Pappa, Luca Romani, Valentino Sacco +5

Sapienza University of Rome, Italy · Inria, Univ. Grenoble Alpes, CNRS, LJK

Efficient Inference for World Action ModelsAI Agent SafetyLanguage Model-Based Control
AskPDF
Mar 20, 2026

PowerLens: Taming LLM Agents for Safe and Personalized Mobile Power Management

Battery life remains a critical challenge for mobile devices, yet existing power management mechanisms rely on static rules or coarse-grained heuristics that ignore user activities and personal preferences. We present PowerLens, a system that tames the reasoning power of Large Language Models (LLMs) for safe and personalized mobile power management on Android devices. The key idea is that LLMs' commonsense reasoning can bridge the semantic gap between user activities and system parameters, enabling zero-shot, context-aware policy generation that adapts to individual preferences through implicit feedback. PowerLens employs a multi-agent architecture that recognizes user context from UI semantics and generates holistic power policies across 18 device parameters. A PDL-based constraint framework verifies every action before execution, while a two-tier memory system learns individualized preferences from implicit user overrides through confidence-based distillation, requiring no explicit configuration and converging within 3--5 days. Extensive experiments on a rooted Android device show that PowerLens achieves 81.7% action accuracy and 38.8% energy saving over stock Android, outperforming rule-based and LLM-based baselines, with high user satisfaction, fast preference convergence, and strong safety guarantees, with the system itself consuming only 0.5% of daily battery capacity.

Xingyu Feng, Chang Sun, Yuzhu Wang +5

China University of Geosciences (Beijing) · Shenzhen University · Hong Kong University of Science and Technology +1

Language Model-Based ControlUser Preference Modeling
AskPDF
Mar 19, 2026

Robotic Agentic Platform for Intelligent Electric Vehicle Disassembly

Electric vehicles (EV) create an urgent need for scalable battery recycling, yet disassembly of EV battery packs remains largely manual due to high design variability. We present our Robotic Agentic Platform for Intelligent Disassembly (RAPID), designed to investigate perception-driven manipulation, flexible automation, and AI-assisted robot programming in realistic recycling scenarios. The system integrates a gantry-mounted industrial manipulator, RGB-D perception, and an automated nut-running tool for fastener removal on a full-scale EV battery pack. An open-vocabulary object detection pipeline achieves 0.9757 mAP50, enabling reliable identification of screws, nuts, busbars, and other components. We experimentally evaluate (n=204) three one-shot fastener removal strategies: taught-in poses (97% success rate, 24 min duration), one-shot vision execution (57%, 29 min), and visual servoing (83%, 36 min), comparing success rate and disassembly time for the battery's top cover fasteners. To support flexible interaction, we introduce agentic AI specifications for robotic disassembly tasks, allowing LLM agents to translate high-level instructions into robot actions through structured tool interfaces and ROS services. We evaluate SmolAgents with GPT-4o-mini and Qwen 3.5 9B/4B on edge hardware. Tool-based interfaces achieve 100% task completion, while automatic ROS service discovery shows 43.3% failure rates, highlighting the need for structured robot APIs for reliable LLM-driven control. This open-source platform enables systematic investigation of human-robot collaboration, agentic robot programming, and increasingly autonomous disassembly workflows, providing a practical foundation for research toward scalable robotic battery recycling.

Zachary Allen, Max Conway, Lyle Antieau +2

Department of Computer Science, University of Colorado, Boulder.

Robotic ManipulationLanguage Model-Based Control
AskPDF
Mar 19, 2026

WASD: Locating Critical Neurons as Sufficient Conditions for Explaining and Controlling LLM Behavior

Precise behavioral control of large language models (LLMs) is critical for complex applications. However, existing methods often incur high training costs, lack natural language controllability, or compromise semantic coherence. To bridge this gap, we propose WASD (unWeaving Actionable Sufficient Directives), a novel framework that explains model behavior by identifying sufficient neural conditions for token generation. Our method represents candidate conditions as neuron-activation predicates and iteratively searches for a minimal set that guarantees the current output under input perturbations. Experiments on SST-2 and CounterFact with the Gemma-2-2B model demonstrate that our approach produces explanations that are more stable, accurate, and concise than conventional attribution graphs. Moreover, through a case study on controlling cross-lingual output generation, we validated the practical effectiveness of WASD in controlling model behavior.

Haonan Yu, Junhao Liu, Zhenyu Yan +2

Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education School of Computer Science, Peking University, Beijing, China

LLM InterpretabilityLanguage Model-Based ControlMechanistic InterpretabilityCausal Interventions in Language Models
AskPDF
Mar 17, 2026

Proactive Rejection and Grounded Execution: A Dual-Stage Intent Analysis Paradigm for Safe and Efficient AIoT Smart Homes

As Large Language Models (LLMs) transition from information providers to embodied agents in the Internet of Things (IoT), they face significant challenges regarding reliability and interaction efficiency. Direct execution of LLM-generated commands often leads to entity hallucinations (e.g., trying to control non-existent devices). Meanwhile, existing iterative frameworks (e.g., SAGE) suffer from the Interaction Frequency Dilemma, oscillating between reckless execution and excessive user questioning. To address these issues, we propose a Dual-Stage Intent-Aware (DS-IA) Framework. This framework separates high-level user intent understanding from low-level physical execution. Specifically, Stage 1 serves as a semantic firewall to filter out invalid instructions and resolve vague commands by checking the current state of the home. Stage 2 then employs a deterministic cascade verifier-a strict, step-by-step rule checker that verifies the room, device, and capability in sequence-to ensure the action is actually physically possible before execution. Extensive experiments on the HomeBench and SAGE benchmarks demonstrate that DS-IA achieves an Exact Match (EM) rate of 58.56% (outperforming baselines by over 28%) and improves the rejection rate of invalid instructions to 87.04%. Evaluations on the SAGE benchmark further reveal that DS-IA resolves the Interaction Frequency Dilemma by balancing proactive querying with state-based inference. Specifically, it boosts the Autonomous Success Rate (resolving tasks without unnecessary user intervention) from 42.86% to 71.43%, while maintaining high precision in identifying irreducible ambiguities that truly necessitate human clarification. These results underscore the framework's ability to minimize user disturbance through accurate environmental grounding.

Xinxin Jin, Zhengwei Ni, Zhengguo Sheng +1

School of Information and Electronic Engineering (Sussex Artificial Intelligence Institute), Zhejiang Gongshang University, Hangzhou 310018, China. · Sussex Artificial Intelligent Institute, Zhejiang Gongshang University, Hangzhou 310018, China · Department of Engineering and Design, University of Sussex, BN1 9RH, Brighton, U.K. +3

Language Model-Based ControlAI Agent ReliabilityAI Agent SafetyRuntime Enforcement for AI AgentsHuman-in-the-Loop AI
AskPDF
Mar 11, 2026

Reflection-Driven Self-Improving Language Model Agents for Adaptive RAN Slicing

Adaptive control in AI-native networks remains challenging because tasks such as dynamic Radio Access Network (RAN) slicing require persistent trade-off decisions among spectrum efficiency, service quality, and reconfiguration stability. Conventional Reinforcement Learning (RL) compresses these competing objectives into handcrafted scalar rewards, requiring exhaustive tuning for a balanced trade-off. Language model (LM) agents provide a new path toward adaptive control because they can semantically represent complex preference relations, but prompt-based adaptation struggles to retain long-horizon interaction history due to finite context limits. We propose a Reflection-Driven Self-Improvement (ReDSI) framework that replaces scalar rewards with semantic feedback for continuous policy adaptation. This framework is built upon a Small Language Model (SLM) agent for fast decision-making, coupled with a bi-perspective reflection to transform interaction history into preference data, subsequently distilling long-horizon decision experience into the SLM parameters through preference-based fine-tuning. Evaluated on a dynamic RAN slicing task, our method achieves an optimal trade-off across multiple competing objectives, outperforming standard RL baselines and existing LM-based agents by demonstrating substantially greater sample efficiency. These results highlight a viable paradigm of self-improving LM agents for reward-free multi-objective continuous control in AI-native networks.

Yuanhao Li, Haozhe Wang, Geyong Min +2

Department of Computer Science, Faculty of Environment, Science and Economy, University of Exeter, Exeter, EX4 4QF, UK. · British Telecom, UK.

Self-Improving AgentsLanguage Model-Based ControlContinual Learning for LLM AgentsFine-Tuning
AskPDF
Mar 10, 2026

LLMs for Text-Based Exploration and Navigation Under Partial Observability

Exploration and goal-directed navigation in unknown layouts are central to inspection, logistics, and search-and-rescue. We ask whether large language models (LLMs) can function as text-only controllers under partial observability -- without code execution, tools, or program synthesis. We introduce a reproducible benchmark with oracle localisation in fixed ASCII gridworlds: each step reveals only a local 5×55\times5 window around the agent and the model must select one of UP/RIGHT/DOWN/LEFT. Nine contemporary LLMs ranging from open/proprietary, dense / Mixture of Experts and instruction- vs. reasoning-tuned are evaluated on two tasks across three layouts of increasing difficulty: Exploration (maximising revealed cells) and Navigation (reach the goal on the shortest path). The experimental results are evaluated on quantitative metrics including success rate, efficiency such as normalised coverage and path length vs. oracle as well as qualitative analysis. Reasoning-tuned models reliably complete navigation across all layouts, yet remain less efficient than oracle paths. Few-shot demonstrations in the prompt chiefly help these Reasoning-tuned models by reducing invalid moves and shortening paths, while classic dense instruction models remain inconsistent. We observe characteristic action priors (UP/RIGHT) that can induce looping under partial observability. Overall, training regimen and test-time deliberation predict control ability better than raw parameter count. These findings suggest lightweight hybridisation with classical online planners as a practical route to deployable partial map systems.

Stephan Sandfuchs, Maximilian Melchert, Jörg Frochte

AKIS – Interdisciplinary Institute for Applied AI and Data Science Ruhr, Bochum University of Applied Sciences, Bochum, Germany

Language Model-Based ControlRobot NavigationAI Agent Benchmarks
AskPDF
Mar 10, 2026

Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning

Associative memory has long underpinned the design of sequential models. Beyond recall, humans reason by projecting future states and selecting goal-directed actions, a capability that modern language models increasingly require but do not natively encode. While prior work uses reinforcement learning or test-time training, planning remains external to the model architecture. We formulate reasoning as optimal control and introduce the Test-Time Control (TTC) layer, which performs finite-horizon LQR planning over latent states at inference time, represents a value function within neural architectures, and leverages it as the nested objective to enable planning before prediction. To ensure scalability, we derive a hardware-efficient LQR solver based on a symplectic formulation and implement it as a fused CUDA kernel, enabling parallel execution with minimal overhead. Integrated as an adapter into pretrained LLMs, TTC layers improve mathematical reasoning performance by up to +27.8% on MATH-500 and 2-3x Pass@8 improvements on AMC and AIME, demonstrating that embedding optimal control as an architectural component provides an effective and scalable mechanism for reasoning beyond test-time training.

Peihao Wang, Shan Yang, Xijun Wang +8

Amazon · The University of Texas at Austin · Georgia Institute of Technology +2

LLM ReasoningLanguage Model-Based ControlFeedback ControlContinuous Latent Reasoning
AskPDF
Mar 9, 2026

Large Language Model-Assisted Superconducting Qubit Experiments

Superconducting circuits have demonstrated significant potential in quantum information processing and quantum sensing. Implementing novel control and measurement sequences for superconducting qubits is often a complex and time-consuming process, requiring extensive expertise in both the underlying physics and the specific hardware and software. In this work, we introduce a framework that leverages a large language model (LLM) to automate qubit control and measurement. Specifically, our framework conducts experiments by generating and invoking schema-less tools on demand via a knowledge base on instrumental usage and experimental procedures. We showcase this framework with two experiments: an autonomous resonator characterization and a direct reproduction of a quantum non-demolition (QND) characterization of a superconducting qubit from literature. This framework enables rapid deployment of standard control-and-measurement protocols and facilitates implementation of novel experimental procedures, offering a more flexible and user-friendly paradigm for controlling complex quantum hardware.

Shiheng Li, Jacob M. Miller, Phoebe J. Lee +14

Department of Physics, University of Chicago, Chicago, IL 60637, USA · Pritzker School of Molecular Engineering, University of Chicago, Chicago, IL 60637, USA · Pico group, QTF Centre of Excellence, Department of Applied Physics, Aalto University School of Science, P.O. Box 13500, 00076 Aalto, Finland +2

Autonomous ExperimentationLanguage Model-Based ControlLLM Tool Use
AskPDF
Mar 5, 2026

RACAS: Controlling Diverse Robots With a Single Agentic System

Many robotic platforms expose an API through which external software can command their actuators and read their sensors. However, transitioning from these low-level interfaces to high-level autonomous behaviour requires a complicated pipeline, whose components demand distinct areas of expertise. Existing approaches to bridging this gap either require retraining for every new embodiment or have only been validated across structurally similar platforms. We introduce RACAS (Robot-Agnostic Control via Agentic Systems), a cooperative agentic architecture in which three LLM/VLM-based modules (Monitors, a Controller, and a Memory Curator) communicate exclusively through natural language to provide closed-loop robot control. RACAS requires only a natural language description of the robot, a definition of available actions, and a task specification; no source code, model weights, or reward functions need to be modified to move between platforms. We evaluate RACAS on several tasks using a wheeled ground robot, a recently published novel multi-jointed robotic limb, and an underwater vehicle. RACAS consistently solved all assigned tasks across these radically different platforms, demonstrating the potential of agentic AI to substantially reduce the barrier to prototyping robotic solutions.

Dylan R. Ashley, Jan Przepióra, Yimeng Chen +5

Center of Excellence in Generative AI, King Abdullah University of Science and Technology (KAUST), Saudi Arabia. · Dalle Molle Institute for Artificial Intelligence Research (IDSIA), Switzerland. · Università della Svizzera italiana (USI), Switzerland. +3

Agentic AI for RoboticsRobotic ControlRoboticsLanguage Model-Based ControlCross-Embodiment Robot Learning
AskPDF
Mar 3, 2026

From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?

In order to flexibly act in an everyday environment, a robotic agent needs a variety of cognitive capabilities that enable it to reason about plans and perform execution recovery. Large language models (LLMs) have been shown to demonstrate emergent cognitive aspects, such as reasoning and language understanding; however, the ability to control embodied robotic agents requires reliably bridging high-level language to low-level functionalities for perception and control. In this paper, we investigate the extent to which an LLM can serve as a core component for planning and execution reasoning in a cognitive robot architecture. For this purpose, we propose a cognitive architecture in which an agentic LLM serves as the core component for planning and reasoning, while components for working and episodic memories support learning from experience and adaptation. An instance of the architecture is then used to control a mobile manipulator in a simulated household environment, where environment interaction is done through a set of high-level tools for perception, reasoning, navigation, grasping, and placement, all of which are made available to the LLM-based agent. We evaluate our proposed system on two household tasks (object placement and object swapping), which evaluate the agent's reasoning, planning, and memory utilisation. The results demonstrate that the LLM-driven agent can complete structured tasks and exhibits emergent adaptation and memory-guided planning, but also reveal significant limitations, such as hallucinations about the task success and poor instruction following by refusing to acknowledge and complete sequential tasks. These findings highlight both the potential and challenges of employing LLMs as embodied cognitive controllers for autonomous robots.

Shinas Shaji, Fabian Huppertz, Alex Mitrevski +1

Institute of AI and Autonomous Systems (A2S), Hochschule Bonn-Rhein-Sieg, Sankt Augustin, Germany · Fraunhofer Institute for Intelligent Analysis and Information Systems, Sankt Augustin, Germany

Large Language Model-Based Robot PlanningLanguage Model-Based ControlCognitive Architectures for AI AgentsEmbodied ReasoningMemory-Augmented Language Model Agents
AskPDF
Mar 1, 2026

DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Driving

Large language models (LLMs) have shown great promise for autonomous driving. However, discretizing numbers into tokens limits precise numerical reasoning, fails to reflect the positional significance of digits in the training objective, and makes it difficult to achieve both decoding efficiency and numerical precision. These limitations affect both the processing of sensor measurements and the generation of precise control commands, creating a fundamental barrier for deploying LLM-based autonomous driving systems. In this paper, we introduce DriveCode, a novel numerical encoding method that represents numbers as dedicated embeddings rather than discrete text tokens. DriveCode employs a number projector to map numbers into the language model's hidden space, enabling seamless integration with visual and textual features in a unified multimodal sequence. Evaluated on OmniDrive, DriveGPT4, and DriveGPT4-V2 datasets, DriveCode demonstrates superior performance in trajectory prediction and control signal generation, confirming its effectiveness for LLM-based autonomous driving systems.

Zhiye Wang, Yanbo Jiang, Rui Zhou +5

School of Information Science and Engineering, Lanzhou University, Lanzhou, China · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University, Beijing, China · The School of Vehicle and Mobility, Tsinghua University, Beijing, China +2

Autonomous DrivingNumerical Reasoning in Language ModelsLanguage Model-Based Control
AskPDF
Feb 28, 2026

Wireless Power Control Based on Large Language Models

This paper investigates the power control problem in wireless networks by repurposing pre-trained large language models (LLMs) as relational reasoning backbones. In hyper-connected interference environments, traditional optimization methods face high computational cost, while standard message passing neural networks suffer from aggregation bottlenecks that can obscure critical high-interference structures. In response, we propose PC-LLM, a physics-informed framework that augments a pre-trained LLM with an interference-aware attention bias. The proposed bias tuning mechanism injects the physical channel gain matrix directly into the self-attention scores, enabling explicit fusion of wireless topology with pre-trained relational priors without retraining the backbone from scratch. Extensive experiments demonstrate that PC-LLM consistently outperforms both traditional optimization methods and state-of-the-art graph neural network baselines, while exhibiting exceptional zero-shot generalization to unseen environments. We further observe that topology-relevant relational reasoning is concentrated in shallow layers, whereas deeper layers encode task-irrelevant semantic noise. Motivated by this finding, we develop a lightweight adaptation strategy that reduces model depth by 50%, significantly lowering inference cost while preserving state-of-the-art spectral efficiency.

Jiacheng Wang, Yucheng Sheng, Le Liang +2

School of Information Science and Engineering, Southeast University, Nanjing 210096, China · Department of Electrical and Computer Engineering, University of California, Santa Cruz, CA 95064, USA

Wireless Resource AllocationLanguage Model-Based ControlLLM Reasoning with Graphs
AskPDF
Feb 27, 2026

SAGE-LLM: Towards Safe and Generalizable LLM Controller with Fuzzy-CBF Verification and Graph-Structured Knowledge Retrieval for UAV Decision

In UAV dynamic decision, complex and variable hazardous factors pose severe challenges to the generalization capability of algorithms. Despite offering semantic understanding and scene generalization, Large Language Models (LLM) lack domain-specific UAV control knowledge and formal safety assurances, restricting their direct applicability. To bridge this gap, this paper proposes a train-free two-layer decision architecture based on LLMs, integrating high-level safety planning with low-level precise control. The framework introduces three key contributions: 1) A fuzzy Control Barrier Function verification mechanism for semantically-augmented actions, providing provable safety certification for LLM outputs. 2) A star-hierarchical graph-based retrieval-augmented generation system, enabling efficient, elastic, and interpretable scene adaptation. 3) Systematic experimental validation in pursuit-evasion scenarios with unknown obstacles and emergent threats, demonstrating that our SAGE-LLM maintains performance while significantly enhancing safety and generalization without online training. The proposed framework demonstrates strong extensibility, suggesting its potential for generalization to broader embodied intelligence systems and safety-critical control domains.

Wenzhe Zhao, Yang Zhao, Ganchao Liu +4

School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, China · Institute of Artificial Intelligence (TeleAI), China Telecom, China

Language Model-Based ControlLLM Agent SafetyAerial RoboticsControl Barrier FunctionsRetrieval-Augmented Generation
AskPDF
Feb 25, 2026

Dynamic Personality Adaptation in Large Language Models via State Machines

The inability of Large Language Models (LLMs) to modulate their personality expression in response to evolving dialogue dynamics hinders their performance in complex, interactive contexts. We propose a model-agnostic framework for dynamic personality simulation that employs state machines to represent latent personality states, where transition probabilities are dynamically adapted to the conversational context. Part of our architecture is a modular pipeline for continuous personality scoring that evaluates dialogues along latent axes while remaining agnostic to the specific personality models, their dimensions, transition mechanisms, or LLMs used. These scores function as dynamic state variables that systematically reconfigure the system prompt, steering behavioral alignment throughout the interaction.We evaluate this framework by operationalizing the Interpersonal Circumplex (IPC) in a medical education setting. Results demonstrate that the system successfully adapts its personality state to user inputs, but also influences user behavior, thereby facilitating de-escalation training. Notably, the scoring pipeline maintains comparable precision even when utilizing lightweight, fine-tuned classifiers instead of large-scale LLMs. This work demonstrates the feasibility of modular, personality-adaptive architectures for education, customer support, and broader human-computer interaction.

Leon Pielage, Ole Hätscher, Mitja Back +2

Institute for Geoinformatics, University of Münster, 48149 Münster, Germany · Faculty of Mathematics and Computer Science, University of Münster, 48149 Münster, Germany · Department of Psychology, University of Münster, 48149 Münster, Germany +1

Medical EducationPersonality Modeling in Language ModelsLarge Language Model-Based Social SimulationLarge Language Model-Based Role-Play SimulationLanguage Model-Based Control
AskPDF
Feb 25, 2026

Two-Stage Active Distribution Network Voltage Control via LLM-RL Collaboration: A Hybrid Knowledge-Data-Driven Approach

The growing integration of distributed photovoltaics (PVs) into active distribution networks (ADNs) has exacerbated operational challenges, making it imperative to coordinate diverse equipment to mitigate voltage violations and enhance power quality. Although existing data-driven approaches have demonstrated effectiveness in the voltage control problem, they often require extensive trial-and-error exploration and struggle to incorporate heterogeneous information, such as day-ahead forecasts and semantic-based grid codes. Considering the operational scenarios and requirements in real-world ADNs, in this paper, we propose a hybrid knowledge-data-driven approach that leverages dynamic collaboration between a large language model (LLM) agent and a reinforcement learning (RL) agent to achieve two-stage voltage control. In the day-ahead stage, the LLM agent receives coarse region-level forecasts and generates scheduling strategies for on-load tap changer (OLTC) and shunt capacitors (SCs) to regulate the overall voltage profile. Then in the intra-day stage, based on accurate node-level measurements, the RL agent refines terminal voltages by deriving reactive power generation strategies for PV inverters. On top of the LLM-RL collaboration framework, we further propose a self-evolution mechanism for the LLM agent and a pretrain-finetune pipeline for the RL agent, effectively enhancing and coordinating the policies for both agents. The proposed approach not only aligns more closely with practical operational characteristics but also effectively utilizes the inherent knowledge and reasoning capabilities of the LLM agent, significantly improving training efficiency and voltage control performance. Comprehensive comparisons and ablation studies demonstrate the effectiveness of the proposed method.

Xu Yang, Chenhui Lin, Xiang Ma +4

Sichuan Energy Internet Research Institute, Tsinghua University, Chengdu 610213, China. · State Grid Zhejiang Electric Power Co., Ltd., Jinhua Power Supply Company, Jinhua 321017, China.

Power SystemsRL ControlLanguage Model-Based Control
AskPDF
Feb 23, 2026

Large Language Model-Assisted UAV Operations and Communications: A Multifaceted Survey and Tutorial

Uncrewed Aerial Vehicles (UAVs) are widely deployed across diverse applications due to their mobility and agility. Recent advances in Large Language Models (LLMs) offer a transformative opportunity to enhance UAV intelligence beyond conventional optimization-based and learning-based approaches. By integrating LLMs into UAV systems, advanced environmental understanding, swarm coordination, mobility optimization, and high-level task reasoning can be achieved, thereby allowing more adaptive and context-aware aerial operations. This survey systematically explores the intersection of LLMs and UAV technologies and proposes a unified framework that consolidates existing architectures, methodologies, and applications for UAVs. We first present a structured taxonomy of LLM adaptation techniques for UAVs, including pretraining, fine-tuning, Retrieval-Augmented Generation (RAG), and prompt engineering, along with key reasoning capabilities such as Chain-of-Thought (CoT) and In-Context Learning (ICL). We then examine LLM-assisted UAV communications and operations, covering navigation, mission planning, swarm control, safety, autonomy, and network management. After that, the survey further discusses Multimodal LLMs (MLLMs) for human-swarm interaction, perception-driven navigation, and collaborative control. Finally, we address ethical considerations, including bias, transparency, accountability, and Human-in-the-Loop (HITL) strategies, and outline future research directions. Overall, this work positions LLM-assisted UAVs as a foundation for intelligent and adaptive aerial systems.

Yousef Emami, Hao Zhou, Radha Reddy +5

Language Model-Based ControlUAV NavigationLarge Language Model-Based Robot Planning
AskPDF
Feb 19, 2026

Agentic Wireless Communication for 6G: Intent-Aware and Continuously Evolving Physical-Layer Intelligence

As 6G wireless systems evolve, growing functional complexity and diverse service demands are driving a shift from rule-based control to intent-driven autonomous intelligence. User requirements are no longer captured by a single metric (e.g., throughput or reliability), but by multi-dimensional objectives such as latency sensitivity, energy preference, computational constraints, and service-level requirements. These objectives may also change over time due to environmental dynamics and user-network interactions. Therefore, accurate understanding of both the communication environment and user intent is critical for autonomous and sustainably evolving 6G communications. Large language models (LLMs), with strong contextual understanding and cross-modal reasoning, provide a promising foundation for intent-aware network agents. Compared with rule-driven or centrally optimized designs, LLM-based agents can integrate heterogeneous information and translate natural-language intents into executable control and configuration decisions. Focusing on a closed-loop pipeline of intent perception, autonomous decision making, and network execution, this paper investigates agentic AI for the 6G physical layer and its realization pathways. We review representative physical-layer tasks and their limitations in supporting intent awareness and autonomy, identify application scenarios where agentic AI is advantageous, and discuss key challenges and enabling technologies in multimodal perception, cross-layer decision making, and sustainable optimization. Finally, we present a case study of an intent-driven link decision agent, termed AgenCom, which adaptively constructs communication links under diverse user preferences and channel conditions.

Zhaoyang Li, Xingzhi Jin, Junyu Pan +2

College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, China.

Wireless Communications6G NetworksIntent-Based NetworkingLanguage Model-Based Control
AskPDF
Feb 15, 2026

Toward Autonomous O-RAN: A Multi-Scale Agentic AI Framework for Real-Time Network Control and Management

Open Radio Access Networks (O-RAN) promise flexible 6G network access through disaggregated, software-driven components and open interfaces, but this programmability also increases operational complexity. Multiple control loops coexist across the service management layer and RAN Intelligent Controller (RIC), while independently developed control applications can interact in unintended ways. In parallel, recent advances in generative Artificial Intelligence (AI) are enabling a shift from isolated AI models toward agentic AI systems that can interpret goals, coordinate multiple models and control functions, and adapt their behavior over time. This article proposes a multi-scale agentic AI framework for O-RAN that organizes RAN intelligence as a coordinated hierarchy across the Non-Real-Time (Non-RT), Near-Real-Time (Near-RT), and Real-Time (RT) control loops: (i) A Large Language Model (LLM) agent in the Non-RT RIC translates operator intent into policies and governs model lifecycles. (ii) Small Language Model (SLM) agents in the Near-RT RIC execute low-latency optimization and can activate, tune, or disable existing control applications; and (iii) Wireless Physical-layer Foundation Model (WPFM) agents near the distributed unit provide fast inference close to the air interface. We describe how these agents cooperate through standardized O-RAN interfaces and telemetry. Using a proof-of-concept implementation built on open-source models, software, and datasets, we demonstrate the proposed agentic approach in two representative scenarios: robust operation under non-stationary conditions and intent-driven slice resource control.

Hojjat Navidan, Mohammad Cheraghinia, Jaron Fontaine +5

IDLab, Department of Information Technology at Ghent University - imec, Ghent, Belgium · Department of Electrical and Computer Engineering, Princeton University, USA

Multi-Agent OrchestrationLanguage Model-Based ControlNetwork SlicingWireless CommunicationsOpen Radio Access NetworkIntent-Based Networking+1
AskPDF
Jan 28, 2026

Demonstration-Free Robotic Control via LLM Agents

Robotic manipulation has increasingly adopted vision-language-action (VLA) models, which achieve strong performance but typically require task-specific demonstrations and fine-tuning, and often generalize poorly under domain shift. We investigate whether general-purpose large language model (LLM) agent frameworks, originally developed for software engineering, can serve as an alternative control paradigm for embodied manipulation. We introduce FAEA (Frontier Agent as Embodied Agent), which applies an LLM agent framework directly to embodied manipulation without modification. Using the same iterative reasoning that enables software agents to debug code, FAEA enables embodied agents to reason through manipulation strategies. We evaluate an unmodified frontier agent, Claude Agent SDK, across the LIBERO, ManiSkill3, and MetaWorld benchmarks. With privileged environment state access, FAEA achieves success rates of 84.9%, 85.7%, and 96%, respectively. This level of task success approaches that of VLA models trained with less than 100 demonstrations per task, without requiring demonstrations or fine-tuning. With one round of human feedback as an optional optimization, performance increases to 88.2% on LIBERO. This demonstration-free capability has immediate practical value: FAEA can autonomously explore novel scenarios in simulation and generate successful trajectories for training data augmentation in embodied learning. Our results indicate that general-purpose agents are sufficient for a class of manipulation tasks dominated by deliberative, task-level planning. This opens a path for robotics systems to leverage actively maintained agent infrastructure and benefit directly from ongoing advances in frontier models. Code is available at https://github.com/robiemusketeer/faea-sim

Brian Y. Tsui, Alan Y. Fang, Tiffany J. Hwu

All authors are independent researchers.

Robotic ControlLanguage Model-Based ControlLarge Language Model-Based Robot PlanningAI ControlRobotic ManipulationAgentic AI for Robotics
AskPDF
Jan 14, 2026

One Prompt, Many Sounds: Modeling Listener Variability in LLM-Based Equalization

Conventional audio equalization is a static process that requires manual and cumbersome adjustments to adapt to changing listening contexts (e.g., mood, location, or social setting). In this paper, we introduce a Large Language Model (LLM)-based alternative that maps natural language text prompts to equalization settings. This enables a conversational approach to sound system control. By utilizing data collected from a controlled listening experiment, our models exploit in-context learning and parameter-efficient fine-tuning techniques to reliably align with population-preferred equalization settings. Our evaluation methods, which leverage distributional metrics that capture users' varied preferences, show statistically significant improvements in distributional alignment over random sampling and static preset baselines. These results indicate that LLMs could function as ``artificial equalizers," contributing to the development of more accessible, context-aware, and expert-level audio tuning methods.

Ioannis Stylianou, Jon Francombe, Pablo Martinez-Nuevo +2

Bang & Olufsen A/S, Struer, Denmark · Department of Electronic Systems, Aalborg University, Aalborg, Denmark · Pioneer Centre for AI, Copenhagen, Denmark

Language Model-Based Control
AskPDF
Dec 29, 2025

Agentic Physical AI toward a Domain-Specific Foundation Model for Energy Systems: A Case Study on Nuclear Reactor Control

The prevailing paradigm in AI for physical systems: scaling general-purpose foundation models toward universal multimodal reasoning, confronts a barrier at the control interface. Frontier vision-language models achieve only 50-53% accuracy on basic quantitative physics tasks, behaving as approximate guessers that preserve semantic plausibility while violating physical constraints. Safety-critical control demands outcome-space guarantees over executed actions, not parameter-space imitation. Here we present a pathway toward domain-specific foundation models through compact language models operating as Agentic Physical AI: policy optimization driven by physics-based simulator validation rather than perceptual inference. We train a 360M-parameter model on synthetic nuclear reactor scenarios scaled from 10^3 to 10^5 examples. Scaling produces strong, regime-dependent reliability gains under nominal simulated conditions, with variance collapse of approximately 500x and elimination of >10% terminal-power excursions on the sampled distribution. Despite balanced exposure to four actuation families, the model concentrates 95% of runtime execution on a single-bank strategy, without reinforcement learning or reward engineering. Representations transfer across simulators without architectural change. We position the system as a candidate decision component within a verification, monitoring, and defense-in-depth architecture, not as a stand-alone safety solution: the demonstrated behavior speaks to closed-loop reliability on a single-step task in simulation and does not yet address off-nominal operation, sensor faults, or uncertainty quantification.

Yoon Pyo Lee, Samrendra Roy, Kazuma Kobayashi +5

The Grainger College of Engineering, Nuclear, Plasma & Radiological Engineering, University of Illinois Urbana-Champaign, Urbana, IL, USA · Department of Nuclear Engineering, Hanyang University, Republic of Korea · University of Texas - El Paso, University Ave, El Paso, USA +4

Language Model-Based ControlSafety-Critical Control
AskPDF
Dec 3, 2025

LAMP: Language-Assisted Motion Planning for Controllable Video Generation

Video generation has achieved remarkable progress in visual fidelity and controllability, enabling conditioning on text, layout, or motion. Among these, motion control - specifying object dynamics and camera trajectories - is essential for composing complex, cinematic scenes, yet existing interfaces remain limited. We introduce LAMP that leverages large language models (LLMs) as motion planners to translate natural language descriptions into explicit 3D trajectories for dynamic objects and (relatively defined) cameras. LAMP defines a motion domain-specific language (DSL), inspired by cinematography conventions. By harnessing program synthesis capabilities of LLMs, LAMP generates structured motion programs from natural language, which are deterministically mapped to 3D trajectories. We construct a large-scale procedural dataset pairing natural text descriptions with corresponding motion programs and 3D trajectories. Experiments demonstrate LAMP's improved performance in motion controllability and alignment with user intent compared to state-of-the-art alternatives establishing the first framework for generating both object and camera motions directly from natural language specifications. Code, models and data are available on our project page.

Muhammed Burak Kizil, Enes Sanli, Niloy J. Mitra +3

Koç University · University College London · Adobe Research +1

Camera-Controlled Video GenerationTrajectory GenerationLanguage Model-Based ControlMotion PlanningControllable Video GenerationLanguage-Conditioned Robot Planning
AskPDF
Nov 14, 2025

GraphPilot: Grounded Scene Graph Conditioning for Language-Based Autonomous Driving

Vision-language models have recently emerged as promising planners for autonomous driving, where success hinges on reasoning over spatial structure and dynamic interactions from multimodal input. However, existing models are typically trained without supervision that explicitly encodes relational dependencies, requiring them to infer interactions between traffic entities implicitly from raw sensor data. In this work, we introduce GraphPilot, a model-agnostic approach that conditions language-based driving models on structured relational context in the form of traffic scene graphs. We serialize scene graphs at different abstraction levels and formats and inject them through structured prompts without modifying the underlying architecture or training objective. Extensive closed-loop evaluations on LangAuto and Bench2Drive show persistent improvements across LMDrive, BEVDriver, and SimLingo. Controlled ablations and hidden-state probing further indicate that relational supervision contributes substantially to these improvements while increasing the decodability of scene relations. Open-loop SpaceDrive results on nuScenes further demonstrate transfer to real-world data. These results indicate that diverse architectures can effectively internalize and ground relational priors through scene graph-conditioned training, even without requiring scene graph input at test-time. Code, fine-tuned models, and our scene graph dataset are publicly available at https://github.com/iis-esslingen/GraphPilot.

Fabian Schmidt, Markus Enzweiler, Abhinav Valada

Esslingen University of Applied Sciences · University of Freiburg

Vision-Language ModelsVLMs for Autonomous DrivingLanguage Model-Based ControlAutonomous DrivingAutonomous Driving PlanningGraph Reasoning
AskPDF
May 10, 2025

LLM-Flock: LLM-Driven Decentralized Multi-Robot Flocking via Influence-Based Consensus with Collision Avoidance

Large Language Models (LLMs) have shown strong capabilities in reasoning and planning, prompting researchers to explore their use as decentralized decision-makers for multi-robot formation control. However, directly applying LLMs to such tasks often leads to unstable behavior, in which robots may collapse to the centroid of their positions or diverge entirely, due to hallucinated reasoning and limited coordination awareness. We propose LLM-Flock, a decentralized multi-robot flocking framework that addresses these challenges through two key mechanisms: (1) an influence-based consensus protocol, where robots iteratively refine individually generated formation plans by adopting those of more connected neighbors, driving the team toward a unified plan without centralized control, and (2) a two-layer collision avoidance strategy that pairs LLM-based urgency reasoning with a programmatic collision checker and re-prompting mechanism. We evaluate LLM-Flock across five state-of-the-art LLMs, including both closed-source (GPT-5, Claude 4.5) and open-source (Llama3.1-405b, Qwen-Max, DeepSeek-R1) models, demonstrating notable improvements in formation stability and convergence over prior LLM-based methods. Real-world experiments with Crazyflie drones further validate the framework's applicability to physical multi-robot systems.

Peihan Li, Lifeng Zhou

Drexel University, United States

Multi-Robot Formation ControlSwarm RoboticsLanguage Model-Based ControlMulti-Agent Consensus
AskPDF
Apr 24, 2025

Exploring Context-aware and LLM-driven Locomotion for Immersive Virtual Reality

Locomotion plays a crucial role in shaping the user experience within virtual reality environments. In particular, hands-free locomotion offers a valuable alternative by supporting accessibility and freeing users from reliance on handheld controllers. To this end, traditional speech-based methods often depend on rigid command sets, limiting the naturalness and flexibility of interaction. In this study, we propose a novel locomotion technique powered by large language models (LLMs), which allows users to navigate virtual environments using natural language with contextual awareness. We evaluate three locomotion methods: controller-based teleportation, voice-based steering, and our language model-driven approach. Our evaluation combines eye-tracking data analysis, including exploratory explainable machine learning analysis with SHAP, and standardized questionnaires (SUS, IPQ, CSQ-VR, NASA-TLX) to examine user experience through both objective gaze-based measures and subjective self-reports of usability, presence, cybersickness, and cognitive load. Our findings show no statistically significant differences in usability, presence, or cybersickness between LLM-driven locomotion and established methods such as teleportation, suggesting its potential as a viable, natural language-based, hands-free alternative. In addition, eye-tracking analysis revealed patterns suggesting tendency toward increased user attention and engagement in the LLM-driven condition. Complementary to these findings, exploratory SHAP analysis revealed that fixation, saccade, and pupil-related features vary across techniques, indicating distinct patterns of visual attention and cognitive processing. Overall, we state that our method can facilitate hands-free locomotion in virtual spaces, especially in supporting accessibility.

Suleyman Ozdel, Kadir Burak Buldu, Enkelejda Kasneci +1

Human-Centered Technologies for Learning, Technical University of Munich, Germany · Munich Center for Machine Learning, Munich, Germany

Human-Computer InteractionLanguage Model-Based Control
AskPDF
Mar 4, 2025

Evolutionary Guided Decoding: Iterative Value Refinement for LLMs

While guided decoding, especially value-guided methods, has emerged as a cost-effective alternative for controlling language model outputs without re-training models, its effectiveness is limited by the accuracy of the value function. We identify that this inaccuracy stems from a core distributional gap: existing methods train static value functions on trajectories sampled exclusively from the base policy, which inherently confines their training to a narrow and suboptimal view of the potential output space. We propose Iterative Value Refinement, a evolutionary framework designed to narrow this gap. It employs Value Exploration to provide a more comprehensive and robust training signal, complemented by Iterative Self-Refinement, which uses the improved value function from one iteration to guide the generation of higher-quality data for the next. Extensive experiments on text summarization, multi-turn dialogue, and instruction following demonstrate the effectiveness of our framework in aligning language models. Our approach not only achieves alignment but also significantly reduces computational costs by leveraging principled value function optimization for efficient and effective control.

Zhenhua Liu, Lijun Li, Ruizhe Chen +5

Shanghai Artificial Intelligence Laboratory · Soochow University · Zhejiang University +1

LLM AlignmentLanguage Model DecodingLanguage Model-Based ControlLarge Language Model-Guided Optimization
AskPDF