Language Model-Based Control

Ask about this topic

Overview

The field studies both how language models can control external systems and how controllers can regulate a model’s own behavior and outputs. Common designs use an LLM for high-level planning, intent interpretation, or code generation, paired with faster controllers such as MPC, reinforcement learning, PID, and deterministic tools; retrieval, world models, and formal safety shields help ground or check proposed actions. Researchers evaluate these systems in settings such as robot manipulation, autonomous driving, traffic signals, networks, HVAC, and software agents, using benchmarks including CARLA, SUMO, LIBERO, nuPlan, and OSWorld as well as some hardware tests. Results show useful planning and adaptation, but reliable deployment remains limited by latency, unsafe or incorrect actions, and gaps between simulation performance and robust real-world control.

Latest papers

CardsList

180 papers29 papers in the last four weeks, up 71% on the four weeks before. 0.2% of all new papers.

  1. Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy

    Oct 26, 2026Yifei Yao, Yude Zou, Guanbo Jiao +12Language Model-Based ControlRobotic Policy Evaluation

    Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Fudan University +1

  2. Adaptive Control of Scientific Instruments with Nonstationary Responses through Multimodal Agents

    Oct 23, 2026Guoyang Shi, Hongwen Li, Yiding Yu +6Adaptive ControlLanguage Model-Based Control

    Beijing ENN Fusion Energy Science and Technology Co., Ltd., Beijing 101111, China · Beijing Key Laboratory of High Magnetic Field Spherical Torus Fusion Energy, Beijing 101111, China · Hebei Key Laboratory of Compact Fusion, Langfang 065000, China

  3. Compiling Autoregressive LLM Control into Policy-Exact Parallel Inference for Hierarchical RAG

    Oct 22, 2026Zehao Lin, Qianfeng Shen, Xixuan Hao +8Language Model-Based ControlEfficient Language Model Inference

    MemTensor (Shanghai) Technology Co., Ltd. · China Haisum Engineering Co., Ltd. · The Hong Kong University of Science and Technology (Guangzhou) +1

  4. Self-Improving Agents with World Model Predictive Paths

    Oct 20, 2026Haori Lu, Tianyu Gao, Yaoyao Liu +1Self-Improving AgentsLLM Agent Planning

    University of Illinois Urbana-Champaign

  5. Frontier LLMs Enter the World of Robotics: A Cross-Model Evaluation on RoboColiseum

    Oct 20, 2026Da Huang, Jichao Wang, Junhui Wu +8Robotic ControlLanguage Model-Based Control

  6. Decision Sufficiency of Learned Representation Interfaces for Safe Generative Control

    Oct 20, 2026Jipeng Li, Yanning ShenLanguage Model-Based ControlLanguage Model Safety Evaluation

    University of California, Irvine

  7. RealControlBench: A Benchmark for Interpretability-Informed Fine-Grained Control of LLMs and Agents

    Oct 19, 2026Xinying Cai, Bin Hu, Mingyu Jin +16Language Model SteeringLanguage Model-Based Control

    University of California, San Diego · Rutgers University · NEC Laboratories America +8

  8. Self-Crystallizing Behavior Trees: Amortizing Foundation-Model Reasoning into Deterministic Control for Autonomous Drones

    Oct 18, 2026Behnam MoradiBehavior TreesRobot Policy Adaptation

    Software Department, KavAI. California, USA

  9. The Bitter Lesson Comes to Robotics

    Oct 18, 2026Christopher Leet, Achyutha Menon, Sravanthi Machcha +3Language Model-Based ControlRobotic Manipulation

    Robocurve · University of Southern California · University of California, Irvine

  10. Blackbox Performance Analysis of LLM-based Network Control

    Oct 17, 2026Mahdi Alizadeh, Pooria Namyar, Arvin Ghavidel +1Black-Box Model AuditingLanguage Model-Based Control

    University of Southern California · Microsoft Research

  11. LocalIntent: Outcome-Specific Structured Generation for Backend-Compatible Smart Building Automation with Local LLMs

    Oct 16, 2026Tianyang Liu, Jiayue Wang, Zhanhao Cheng +4LLM PromptingLanguage Model-Based Control

    Beijing Dublin International College, University College Dublin, Beijing, China · School of Computer Science, University College Dublin, Dublin, Ireland

  12. X-RIC: An Agentic RAN-Intelligent Controller Benchmark for Enhancing Reinforcement Learning

    Oct 15, 2026Adhwaa Al-Chaab, Ayman Younis, Dario PompiliNetwork SlicingAgentic RL

    †Department of Electrical and Computer Engineering, Rutgers University–New Brunswick, NJ, USA

  13. DeciCAD: Harness-Grounded Learning of Action Policies for CAD Agents

    Oct 15, 2026Yufeng Peng, Yixin Zhang, Ziyi Liu +5LLM Agent HarnessesLanguage Model-Based Control

    University of Science and Technology of China · Nanyang Technological University

  14. Multi-System Cognitive Steering for Generative Communication

    Oct 13, 2026Eric Lacosse, Pablo Tano, Daniel C. McNameeLanguage Model SteeringLanguage Model-Based Control

    Champalimaud Centre for Restorative Neurotechnology, Champalimaud Research, Lisbon, Portugal

  15. Learning Adaptive Deliberation Control for Reliable Recognition with Multi-Agent Language Models

    Oct 13, 2026Yuan Liu, Haoyue Peng, Fanliang Hu +2Multi-Agent DeliberationMulti-Agent Debate

    Medical Artificial Intelligence Laboratory, Chongqing University Cancer Hospital, Chongqing University, Chongqing, 400030, China

  16. CIPHEUR: LLM-Guided Structural Search Control for Conflict-Graph Optimization

    Oct 12, 2026Yuqing Cheng, Yan Jia, Yanjie Song +3Language Model-Guided Search HeuristicsCombinatorial Optimization

    National University of Defense Technology, Changsha, China · Dalian Maritime University, Dalian, China · Jiangsu University of Technology, Changzhou, China

  17. Adaptive Code Generation for Controlling Robots

    Oct 7, 2026Justus Flerlage, Thorsten Wittkopp, Alexander Acker +1Code Generation for RoboticsLanguage Model-Based Control

    Technische Universität Berlin, Germany · logsight.ai, Berlin, Germany

  18. XTurnix: Large-Scale Self-Supervised Turn Control through Two-State Binary Decisions

    Oct 3, 2026Zhanxun Liu, Yifan Duan, Hengtao Wu +7Turn-Taking PredictionSpoken Dialogue Systems

    MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University · Shanghai Innovation Institute · Shanghai AI Laboratory +3

  19. LS-AR: Future-Predictive Latent Steering in Autoregressive LLMs

    Oct 2, 2026Anubha Gupta, Eduardo PignatelliAutoregressive Language ModelingLanguage Model-Based Control

    Electronic & Electrical Engineering University College London London, UK

  20. Mimir: Physics-Grounded LLM Agents for Long-Horizon Irrigation Control

    Oct 1, 2026Yimeng Liu, Mi Zhang, Younsuk Dong +1Language Model-Based ControlLLM Agent Self-Improvement

    Michigan State University · Ohio State University

  21. Harness Annealing: Learning to Act with Less External Control

    Oct 1, 2026Yingxuan Yang, Huacan Chai, Ying WenLanguage Model-Based ControlAgent Harness Optimization

    Shanghai Jiao Tong University · University of California, Berkeley

  22. Towards Hierarchical Cyber Defense with Large Language Models: From Planning to Execution

    Sep 30, 2026Harshith Doppalapudi, Nathaniel D. Bastian, Ankit ShahAutonomous Cyber DefenseLanguage Model-Based Control

    Indiana University Bloomington, IN, USA · Johns Hopkins University Baltimore, MD, USA

  23. Toward Controlling Biology with Language:Offline Learning of Prompt-Conditioned Interventions for Cells, Organoids, and Biobots

    Sep 30, 2026Nam H. Le, Douglas Blackiston, Michael Levin +1Language Model-Based Control

    University of Vermont · Tufts University, Allen Discovery Center

  24. Code to Control: Synthesizing Parameterized Reactive Controllers

    Sep 30, 2026Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates +1RL ControlLanguage Model-Based Control

    Harvard University · Massachusetts Institute of Technology · Florida Institute for Human and Machine Cognition

  25. MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptation

    Sep 30, 2026Mehdi Jafari, Hao Xue, Flora SalimLanguage Model SteeringActivation Steering

    School of Computer Science and Engineering, UNSW Sydney, Australia · ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S) · The Hong Kong University of Science and Technology (Guangzhou), China

  26. Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies

    Sep 29, 2026Galbot Team, Xuchuan Chen, Xiaoqian Cheng +31Language Model-Based ControlRobot Foundation Models

    Galbot Team

  27. InstCharVoice: Grounding Natural-Language Instructions for Character-Level Control in Text-to-Speech

    Sep 28, 2026Sihang Nie, Xueru Li, Xiaofen Xing +4TTS SynthesisLanguage Model-Based Control

    South China University of Technology · Huya Inc. · The Hongkong Polytechnic University

  28. Dyad: Extending Large Language Models with Native Typed Decision-Making

    Sep 28, 2026Yundaichuan Zhan, Weishi Wang, Wenbiao Liu +5Language Model-Based ControlRL for LLM Agents

    Zhejiang University · SAP · Central South University +3

  29. A.D.A.M.O. (Agent for language-Driven Actions with Multimodal Observations): A Visual-Symbolic Framework for Virtual Humans

    Sep 28, 2026Alessandro Emmanuel Pecora, Stefano Calzolari, Francesco Strada +1Language Model-Based ControlWorld Models

    DAUIN, Politecnico di Torino, Torino, Italy

  30. Steering Language Model Goals with Value Transplant

    Sep 28, 2026Pengcheng Jiang, Fabien RogerLanguage Model SteeringLanguage Model-Based Control

    Anthropic Fellows Program · Anthropic