Human Instruction
Human instruction following in AI focuses on developing models capable of accurately and reliably executing complex tasks based on diverse instructions, encompassing text, images, and audio. Current research emphasizes improving model alignment through techniques like instruction tuning and response tuning, often utilizing large language models (LLMs) and diffusion transformers, and exploring novel evaluation metrics for multi-modal, multi-turn interactions. This field is crucial for advancing human-computer interaction, enabling more intuitive and effective collaboration between humans and AI systems across various domains, from robotics and manufacturing to healthcare and education.
Papers
Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models
Potsawee Manakul, Guangzhi Sun, Warit Sirichotedumrong, Kasima Tharnpipitchai, Kunat Pipatanakul
SIFToM: Robust Spoken Instruction Following through Theory of Mind
Lance Ying, Jason Xinyu Liu, Shivam Aarya, Yizirui Fang, Stefanie Tellex, Joshua B. Tenenbaum, Tianmin Shu
Behavior Tree Generation using Large Language Models for Sequential Manipulation Planning with Human Instructions and Feedback
Jicong Ao, Yansong Wu, Fan Wu, Sami Haddadin
Overcoming linguistic barriers in code assistants: creating a QLoRA adapter to improve support for Russian-language code writing instructions
C. B. Pronin, A. V. Volosova, A. V. Ostroukh, Yu. N. Strogov
Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments
Zoya Volovikova, Alexey Skrynnik, Petr Kuderov, Aleksandr I. Panov
Soft Prompts Go Hard: Steering Visual Language Models with Hidden Meta-Instructions
Tingwei Zhang, Collin Zhang, John X. Morris, Eugene Bagdasarian, Vitaly Shmatikov