Robot foundation models have substantially advanced perception and control, but natural human-robot collaboration requires more than executing isolated commands. A robot must recognize ambiguity, maintain context across turns, communicate its intentions, and revise ongoing behavior as the user's intent changes. We present Ludi0.1, an agentic system for socially intelligent robots that integrates interactive speech, multimodal reasoning, memory, navigation, and learned manipulation. Its decision-making core is a fine-tuned vision-language model trained on multi-turn interaction traces spanning ambiguous requests, clarifications, corrections, interruptions, mixed social and task dialogue, and multi-step tasks. A purpose-built harness manages the model-tool interaction loop, while specialized navigation and manipulation policies execute physical skills. Ludi0.1 demonstrates a practical path toward fluid human-robot collaboration today while producing the multimodal interaction traces needed to develop a more deeply integrated foundation model for robots and people.
Figures & tables
Figure 1: Social intelligence in action. In this real-robot demonstration on a Unitree G1 humanoid, a user asks Ludi 0.1 to bring a Coke to Chloe in the bedroom, and Ludi 0.1 begins reaching for the Coke. When the user remembers that Chloe prefers Pepsi and changes the request, the robot abandons the original action, retrieves the Pepsi, navigates to the bedroom, and delivers it to Chloe’s desk. The episode illustrates how dialogue, contextual reasoning, navigation, and manipulation are coordinated within a single evolving human–robot interaction.
Figure 2: Architecture of the Ludi 0.1 embodied agent. The VLM provides scene perception, reasoning, dialogue, and tool selection, while the harness manages model calls, tool execution, results, and shared context across turns.
Figure 3: An example of generated data. The user identifies the target by spatial position (“the middle laptop”); the model grounds the target in the four-view observation before issuing a reachability check. The multi-turn tool history ( context ) and the latest four-view observation form the input, while the reasoning and one action ( img + turn ) are the supervision target.
Model
Success (judge-overturned)
GPT-5.6 (GEPA-optimized prompt; 1 )
26/28 (0)
Qwen3.5-4B (vanilla)
10/28 (1)
Qwen3.5-4B (fine-tuned)
17/28 (1)
Qwen3.5-9B (vanilla)
11/28 (0)
Qwen3.5-9B (fine-tuned)
20/28 (1)
Table 1: Interaction Core28 closed-loop scenario benchmark (simulation). Episode-level task success after LLM-judge appeal; the parenthesized count is how many of those successes the judge granted by overturning a language-semantic gate failure. Fine-tuned models use LoRA ( 13 ) applied to the corresponding Qwen3.5 base models.
Figure 4: Example of a real-robot pick-and-place task using the learned VLA policy. The Unitree G1, equipped with a Dex-3 hand, grasps the target can from the tabletop using egocentric visual observations.