This work explores a multimodal approach to the alignment of the behavior of open-ended learning autonomous robots with human purposes through the use of large language models as cultural enablers. While autonomous robots can generate their own skills and goals through interaction with the environment, they often lack mechanisms to ensure that these remain relevant and aligned to those of human users. The proposed approach makes use of the general and cultural knowledge contained in LLMs and VLMs to establish a multimodal interactive alignment process in which human purposes, expressed in natural language, are transformed into motivational components of a robot architecture in the form of robot missions and drives that guide the behavior of the robot. The system is structured into three layers for semantic alignment, mission generation, and drive modeling. This design enables a bidirectional interaction loop: human inputs define high-level purposes, robots expand them into executable missions making use of multimodal information relayed by LLMs and VLMs, and feedback maintains alignment with human intent. Experiments in simulation and with a physical robot demonstrate that the process produces valid missions and behaviors, supporting effective human-agent alignment in interactive scenarios with autonomous robots.
Figures & tables
Fig. 1 : Overview of the proposed framework. The Alignment Engine manages human–robot interaction to align human and robot purposes. The Translation Engine converts aligned purposes into missions and drives, which are integrated into the Motivational System of a cognitive architecture to guide autonomous robot behavior.
Purpose
Example of non-expert user input
Arrange apples by color
“Place the apples where they belong [as in the image]”
Classifying cylinders by color
“Sort the cylinders on the table [as in the image].”
Stacking cylinders in the center of a table from biggest to smallest
“Arrange the pieces on top of each other [as in the image]”
TABLE I : Example ambiguous user inputs for the three evaluated purposes, simulating how non-expert users describe tasks in natural language.
Arrange apples by color
“The robot should arrange the apples on the table by their color. First, the robot has to grasp the red apple (object2) and place in in the right side of the table ( x_position>0.5 ). After that, the robot has to grasp the green apple (object3) and place it on the left side of the table ( x_position<0.5 ).”
Classifying cylinders by color into baskets
“The robot must arrange the cylinders by color into baskets. First it has to reach the red cylinder (object2) with its hand, grasp it and pick it up. Then, the robot should move its hand, with the red cylinder picked, to the green basket (object4) and release it into the basket. After that, the robot should move its hand to the blue cylinder and grasp it. Then, the robot should take the blue cylinder (object3) to the position of the yellow basket (object5) and place the blue cylinder into the basket.”
Stacking cylinders in the center of the table from largest to smallest
“The robot’s task is to stack the cylinders in the center of the table by their size (based on their diameters). The largest cylinder goes at the bottom and the smallest one at the top. To do this, first, the robot should grasp the object4 (largest cylinder), move it to the center of the table. After that, the robot should grasp the object2 (second largest cylinder) and place it on top of the object4. Last, it has to grasp the object3 (smallest cylinder), move it to the center of the table and stack it on top of the object2.”
TABLE II : Detailed ideal purpose descriptions used as reference to estimate the semantic similarity.
Fig. 2 : Box plot of semantic similarity between input purposes and output descriptions generated by the Alignment Engine under multimodal and unimodal configurations, with and without human feedback.
Mission [label, weight]
Drive
[”red_apple_in_hand”, 0.2]
∣gripper−1∣+∣xrob−xred∣+∣yrob−yred∣
[”red_apple_right_side”, 0.4]
max(0,0.5−xred)
[”green_apple_in_hand”, 0.6]
∣gripper−1∣+∣xrob−xgreen∣+∣yrob−ygreen∣
[”green_apple_left_side”, 0.8]
max(0,xgreen−0.5)
TABLE III : Missions and drive functions for the Arrange apples by color ideal purpose.
Fig. 3 : Validity of generated missions across the three tested purposes. The ideal purpose (Table II ) had a validity rate of 100% across the three purposes.
Fig. 4 : Validity of generated drive functions across the three tested purposes and the success rate for the ideal purposes (Table II ).
Fig. 5 : Simulated and real environment used to validate the integration of the generated missions and drives within the e-MDB cognitive architecture.
Fig. 6 : Number of iterations required for the robot to achieve the assigned purpose using the Alignmnet and Translation Engines compared to using just novelty.
Improving embodied reasoning in multimodal-large-language models (MLLMs) is essential for building vision-language-action models (VLAs) on top of them to readily translate multimodal understanding into low-level actions. Accordingly, recent work has explored enhancing embodied reasoning in MLLMs through supervision of vision-question-answering type. However, these approaches have been reported to result in unstable VLA performance, often yielding only marginal or even negative gains. In this paper, we propose a more systematic MLLM training framework RoboAlign that reliably improves VLA performance. Our key idea is to sample action tokens via zero-shot natural language reasoning and refines this reasoning using reinforcement learning (RL) to improve action accuracy. As a result, RoboAlign bridges the modality gap between language and low-level actions in MLLMs, and facilitate knowledge transfer from MLLM to VLA. To validate the effectiveness of RoboAlign, we train VLAs by adding a diffusion-based action head on top of an MLLM backbone and evaluate them on major robotics benchmarks. Remarkably, by performing RL-based alignment after SFT using less than 1% of the data, RoboAlign achieves performance improvements of 17.5%, 18.9%, and 106.6% over SFT baselines on LIBERO, CALVIN, and real-world environments, respectively.
Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.
Robot assistants for older adults and people with disabilities need to perform collaborative tasks with users effectively. The core component of these systems is an interaction manager whose job is to observe and assess the task and infer the state of the human and their intent for the robot to choose the best course of action. Due to the sparseness of the data in this domain, the policy for such multimodal systems is often crafted by hand; as the complexity of interactions grows, this process is not scalable. This paper proposes a reinforcement learning (RL) approach to automatically generate the multimodal policy of the robot. Our system focuses on a realistic scenario where a robot assists a user in locating objects within a home environment, managing multimodal signals, including language and physical actions, to select the best action. In contrast to traditional dialog systems, our agent is trained with a simulator that uses human data and can deal with multiple modalities. We use a simple high-level reward function that needs no fine-tuning and enforce some preconditions to speed up the training process. A human study evaluating the system in a real-world setting demonstrates promising results, indicating high usability and effective task completion. This RL-based approach offers a scalable and interpretable alternative for designing interaction managers in multimodal human-robot collaborations.