Instruction Following

Follow this topic Ask about this topic

Instruction-following research asks whether language models and agents can carry out what users request, including exact formats, multiple constraints, changing requirements, and rules that override other instructions. Researchers test these abilities with automatically checked benchmarks and realistic settings such as long conversations, tool use, coding, speech, and embodied tasks. To improve performance, they use supervised fine-tuning, preference learning and reinforcement learning with verifiable or rubric-based rewards, as well as prompt compilation, model steering, and inference-time checks. Models can handle many simple instructions, but reliability often falls when constraints interact or accumulate, and improving compliance without harming reasoning, task quality, or general capabilities remains a central challenge.

Latest papers

135 papers22 papers in the last four weeks, up 57% on the four weeks before. 0.1% of all new papers.

Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance
May 8, 2026Jiachen Yu, Zhihao Xu, Junjie Wang +1Instruction Following Rubric-Based Evaluation
1Tsinghua University · 2Renmin University of China
SEQUOR: A Multi-Turn Benchmark for Realistic Constraint Following
May 7, 2026Beatriz Canaverde, Duarte M. Alves, José Pombal +2Instruction Following Generative AI Evaluation
Instituto de Telecomunicac¸˜oes · Instituto Superior T´ecnico, Universidade de Lisboa · Sword Health +2
Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning
May 7, 2026Bing Wang, Ximing Li, Changchun Li +3LLM Fine-Tuning Gradient Interference
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering, Ministry of Education, Jilin University · RIKEN Center for Advanced Intelligence Project +1
MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following
May 5, 2026Jaeyun Lee, Junyoung Koh, Zeynel Tok +2LLM Evaluation LLM-as-a-Judge
University of Oxford · Yonsei University
When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models
May 1, 2026Sailesh Panda, Pritam Kadasi, Abhishek Upperwal +1LLM Evaluation Instruction Following
Indian Institute of Technology Gandhinagar · Soket AI
Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation
Apr 30, 2026Garvin KruthofLLM Evaluation LLM Alignment
Technical University of Munich
From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
Apr 30, 2026Qingyu Ren, Tianjun Pan, Xingzhou Chen +1Reward Modeling RL for Language Models
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · 2Shanghai Artificial Intelligence Laboratory
Compliance versus Sensibility: On the Reasoning Controllability in Large Language Models
Apr 29, 2026Xingwei Tan, Marco Valentino, Mahmud Elahi Akhter +3LLM Interpretability Instruction Following
School of Computer Science, University of Sheffield · School of EECS, Queen Mary University of London · The Alan Turing Institute
Beyond the Attention Stability Boundary: Agentic Self-Synthesizing Reasoning Protocols
Apr 27, 2026Dahlia Shehata, Ming LiInstruction Following Attention Mechanisms
University of Waterloo · Canada
Omission Constraints Decay While Commission Constraints Persist in Long-Context LLM Agents
Apr 22, 2026Yeran GamageAI Agent Monitoring LLM Security
University of South Florida · Independent AI Security Researcher
Measuring Distribution Shift in User Prompts and Its Effects on LLM Performance
Apr 19, 2026Parker Seegmiller, Sarah Masud PreumLLM Evaluation Instruction Following
Department of Computer Science Dartmouth College Hanover, NH, USA
TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles
Apr 19, 2026Yirong Zeng, Yufei Liu, Xiao Ding +9Reward Modeling LLM Alignment
Harbin Institute of Technology SCIR Lab · Peking University · Huawei Technologies Co., Ltd
Mini-BEHAVIOR-Gran: Revealing U-Shaped Effects of Instruction Granularity on Language-Guided Embodied Agents
Apr 18, 2026Sukai Huang, Chenyuan Zhang, Fucai Ke +4AI Agent Evaluation AI Agent Benchmarks
Faculty of Information Technology, Monash University
VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions
Apr 12, 2026Hung-Ting Su, Ting-Jun Wang, Jia-Fong Yeh +2VLN Benchmarks Vision-Language Navigation
National Taiwan University · DRIC · National Tsing Hua University
To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models
Apr 8, 2026Ane G. Domingo-Aldama, Iker De La Iglesia, Maitane Urruela +2Medical LLMs Domain Adaptation for LLMs
How LLMs Follow Instructions: Skillful Coordination, Not a Universal Mechanism
Apr 7, 2026Elisabetta Rocchetti, Alfio FerraraLLM Interpretability Instruction Following
Department of Computer Science, Università degli Studi di Milano
Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
Apr 3, 2026Tianze Xu, Yanzhao Zheng, Pengrui Lu +11RL for Language Models Rubric-Based RL
Shanghai Jiao Tong University · GAIR · Alibaba Group +2
AURA: Multimodal Shared Autonomy for Real-World Urban Navigation
Apr 2, 2026Yukai Ma, Honglin He, Selina Song +2Shared Autonomy Robot Navigation
University of California, Los Angeles · Zhejiang University
Countering Catastrophic Forgetting of Large Language Models for Better Instruction Following via Weight-Space Model Merging
Apr 2, 2026Mengxian Lyu, Cheng Peng, Ziyi Chen +3Domain Adaptation for LLMs Medical LLMs
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA · Department of Engineering Education, Herbert Wertheim College of Engineering, University of Florida, Gainesville, FL, USA · Preston A. Wells, Jr. Center for Brain Tumor Therapy, Lillian S. Wells Department of Neurosurgery, University of Florida, Gainesville, Florida, USA
Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights
Mar 30, 2026Eneko Valero, Maria Ribalta i Albado, Oscar Sainz +2Language Model Merging Multilingual Language Models
HiTZ Center - Ixa, University of the Basque Country UPV/EHU
RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following
Mar 26, 2026Tianjun Pan, Xuan Lin, Wenyan Yang +7Rubric-Based Evaluation LLM-as-a-Judge
College of Computer Science and Artificial Intelligence, Fudan University · Ant Group · Donghua University
CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context
Mar 23, 2026Giovana Kerche Bonás, Roseval Malaquias Junior, Marcos Piau +6Instruction Following Multilingual Language Model Evaluation
Maritaca AI · Jusbrasil
TiCo: Time-Controllable Spoken Dialogue Model
Mar 23, 2026Kai-Wei Chang, Wei-Chih Chen, En-Pei Hu +2Spoken Dialogue Systems Instruction Following
MIT · NTU · NTU AI-CoRE
Instruction-Tuned, but Not More Verifiable Instruction-Following: A Cross-Task Diagnosis for LoRA Adapters
Mar 23, 2026Junyi ZouInstruction Tuning Instruction Following
Zjydiary Group
When Contextual Inference Fails: Cancelability in Interactive Instruction Following
Mar 20, 2026Natalia Bila, Kata Naszádi, Alexandra Mayn +1Pragmatic Reasoning in Language Models Instruction Following
Department of Language Science and Technology, Saarland University · Language Technology Lab, University of Amsterdam
Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
Mar 19, 2026Myeongkyun Kang, Soopil Kim, Xiaoxiao Li +1Medical VLMs Large Vision-Language Models
Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada · Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea · Division of Intelligent Robot, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea +2
Evaluating Ill-Defined Tasks in Large Language Models
Mar 17, 2026Yi Zhou, Basel ShbitaLLM Evaluation Instruction Following
IBM Research, San Jose, CA
HATS: Hardness-Aware Trajectory Synthesis for GUI Agents
Mar 12, 2026Rui Shao, Ruize Gao, Bin Xie +5LLM Agent Trajectory Synthesis GUI Automation
Harbin Institute of Technology, Shenzhen · Shenzhen Loop Area Institute · Huawei Noah’s Ark Lab
CCR-Bench: A Comprehensive Benchmark for Evaluating LLMs on Complex Constraints, Control Flows, and Real-World Cases
Mar 9, 2026Xiaona Xue, Yiqiao Huang, Jiacheng Li +9LLM Evaluation Instruction Following
Jiutian Artificial Intelligence Research Institute, China Mobile, Beijing, China
Did You Forget What I Asked? Prospective Memory Failures in Large Language Models
Mar 7, 2026Avni MittalInstruction Following LLM Reliability

May 8, 2026J/K move · Enter open · S save

Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance

Jiachen Yu, Zhihao Xu, Junjie Wang, Yujiu Yang

1Tsinghua University · 2Renmin University of China

Instruction Following Rubric-Based Evaluation Rubric-Guided Generation Text Generation LLM Reasoning

Open paper AskPDF arXiv

Rubrics have been extensively utilized for evaluating unverifiable, open-ended tasks, with recent research incorporating them into reward systems for reinforcement learning. However, existing frameworks typically treat rubrics only as external evaluator disjointed from the policy's primary reasoning trace. Such design confines rubrics to post-hoc measurement, leaving them unable to actively guide the model's generation process. In this work, we introduce Think-with-Rubrics, a novel paradigm for instruction following tasks. Think-with-Rubrics integrates rubric generation into the reasoning context, transforming the rubric from an independent artifact into an internal guidance of LLM's generation. During training, LLM sequentially generates a rubric followed by a response, while a trained rubric verifier provides joint supervision by evaluating the consistency between the answer and the self-generated / golden rubrics. Experiments across multiple benchmarks demonstrate that Think-with-Rubrics consistently outperforms the Rubric-as-Reward baseline supervised by golden rubrics by an average of 3.87 points. We have also discussed the mechanism by which Think-with-Rubrics enhances model performance. Experimental results demonstrate that supervision from golden rubrics and self-generated rubrics enhances the performance of Think-with-Rubrics by improving the quality of self-generated rubrics and increasing the internal consistency of responses respectively.