Robot foundation models have substantially advanced perception and control, but natural human-robot collaboration requires more than executing isolated commands. A robot must recognize ambiguity, maintain context across turns, communicate its intentions, and revise ongoing behavior as the user's intent changes. We present Ludi0.1, an agentic system for socially intelligent robots that integrates interactive speech, multimodal reasoning, memory, navigation, and learned manipulation. Its decision-making core is a fine-tuned vision-language model trained on multi-turn interaction traces spanning ambiguous requests, clarifications, corrections, interruptions, mixed social and task dialogue, and multi-step tasks. A purpose-built harness manages the model-tool interaction loop, while specialized navigation and manipulation policies execute physical skills. Ludi0.1 demonstrates a practical path toward fluid human-robot collaboration today while producing the multimodal interaction traces needed to develop a more deeply integrated foundation model for robots and people.
Figures & tables
Figure 1: Social intelligence in action. In this real-robot demonstration on a Unitree G1 humanoid, a user asks Ludi 0.1 to bring a Coke to Chloe in the bedroom, and Ludi 0.1 begins reaching for the Coke. When the user remembers that Chloe prefers Pepsi and changes the request, the robot abandons the original action, retrieves the Pepsi, navigates to the bedroom, and delivers it to Chloe’s desk. The episode illustrates how dialogue, contextual reasoning, navigation, and manipulation are coordinated within a single evolving human–robot interaction.
Figure 2: Architecture of the Ludi 0.1 embodied agent. The VLM provides scene perception, reasoning, dialogue, and tool selection, while the harness manages model calls, tool execution, results, and shared context across turns.
Figure 3: An example of generated data. The user identifies the target by spatial position (“the middle laptop”); the model grounds the target in the four-view observation before issuing a reachability check. The multi-turn tool history ( context ) and the latest four-view observation form the input, while the reasoning and one action ( img + turn ) are the supervision target.
Model
Success (judge-overturned)
GPT-5.6 (GEPA-optimized prompt; 1 )
26/28 (0)
Qwen3.5-4B (vanilla)
10/28 (1)
Qwen3.5-4B (fine-tuned)
17/28 (1)
Qwen3.5-9B (vanilla)
11/28 (0)
Qwen3.5-9B (fine-tuned)
20/28 (1)
Table 1: Interaction Core28 closed-loop scenario benchmark (simulation). Episode-level task success after LLM-judge appeal; the parenthesized count is how many of those successes the judge granted by overturning a language-semantic gate failure. Fine-tuned models use LoRA ( 13 ) applied to the corresponding Qwen3.5 base models.
Figure 4: Example of a real-robot pick-and-place task using the learned VLA policy. The Unitree G1, equipped with a Dex-3 hand, grasps the target can from the tabletop using egocentric visual observations.
Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.
Foundational models have advanced social robotics, enabling richer perception and communicative interaction with users. However, current systems still struggle with multi-turn engagement, social-relationship reasoning, and contextually grounded dialogue at scale. We present ARIS (Agentic and Relationship Intelligence System), an agentic AI framework that unifies multimodal reasoning, a graph-based Social World Model, and retrieval-augmented generation (RAG) within a single modular architecture for social robots. We evaluate ARIS with the Pepper robot in a robot-mediated dyadic conversational setting, comparing it against a large language model baseline. A user study (N=23) shows that ARIS yields significantly higher perceived intelligence, animacy, anthropomorphism, and likeability. Our contributions are threefold: (1)~a Social World Model that explicitly maps and updates social relationships between users through a knowledge graph, enabling social reasoning and re-identification across encounters; (2)~an efficient RAG-based conversational pipeline that maintains bounded latency as dialogue histories grow to thousands of exchanges while preserving response relevance; and (3)~system integration and empirical validation of these components within a modular agentic architecture that coordinates speech, vision, and physical action through structured APIs. The implementation of ARIS will be released as open source upon publication.
Anticipating whether a person will interact from one's own perspective is a highly intuitive task for humans, that relies on a combination of cues. We investigate how humans perform at predicting a person's intention to interact from a service robot's point of view, using pose-only or full video input, then benchmark different lightweight pose-based models and state-of-the-art vision-language models. We conducted our benchmark on the HUI360 dataset on a fixed pilot subset of 100 test tracks (25 positive, 75 negative). We found that with pose-only input, human annotators outperform lightweight trained pose models but not by large margins (+0.08 in F1-Score). But when given full egocentric video with a target bounding box, human annotators perform substantially better and largely outperform the Vision-Language Models (+0.2 in F1-Score). We also compared VLMs of different size and under different input conditions, and found that the best results do not correlate with model size. Our result confirms that predicting interactions is a challenging task for social robots and that reasoning-capable models are necessary but their actual reasoning capabilities alone do not suffice to match the social intuition of humans.