Embodied agents must translate multimodal observations and user requests into executable actions while adapting to product constraints, changing goals, and execution feedback. We present JoyInside-VLM-Agent, a ReAct-style multimodal agent developed through product-conditioned trajectory construction and staged policy learning for robotic platforms and smart hardware. A closed-loop sandbox generates multimodal interaction trajectories covering goal revisions and failure recovery, which are checked against execution outcomes and semantic criteria. Supervised fine-tuning and multi-teacher on-policy distillation precede reinforcement learning with DPER (Dual-Source Privileged On-Policy Self-Distillation with Token-Level Evidence Routing). DPER routes token-level evidence between a frozen privileged teacher and a privileged student sharing the rollout policy's parameters, modulating reward-derived token advantages while preserving their signs. Training rewards combine generative process evaluation, multi-judge rankings, and task-queue consistency. JoyInside-VLM-Agent achieves a BFCL v4 overall score of 77.44, exceeding the evaluated baselines; results on the other public benchmarks are mixed. On JoyInside-Devices and JoyInside-Robotics, the evaluated system, which includes domain-specific skill refinement, leads the baselines on all reported metrics. Two real-world quadruped case studies illustrate visually grounded search, replanning after execution failure, and explicit failure reporting.
Figures & tables
Figure 1: JoyInside-VLM-Agent overview. Top: Real-world quadruped interactions show navigation with visual reporting (left), greeting (center), and dialogue (right). Bottom: Product-conditioned data construction (Stage 1) supplies trajectories and executable environments for SFT, multi-teacher OPD, and DPER (Stage 2). The right panel summarizes agent capabilities.
Figure 2: Product-conditioned multimodal data construction. (A, steps 1–2) Compatible product modules supply the developer prompt, executable rules, and initial scene. (B, step 3) Routed actions produce user feedback or execution results; state changes determine subsequent visual observations. (Step 4) Deterministic and multimodal audits yield verified demonstrations and preference pairs for reward-model training. Executable environments retain the product rules for online rollouts. Verified demonstrations include appropriate refusals and compliant recovery from environmental failures as well as successful task completion.
Figure 3: Three-stage policy learning. All stages use the shared product-conditioned context. (A) SFT trains on assistant tokens in multimodal demonstrations. (B) Text and vision teachers supervise student-generated prefixes through reverse-KL OPD. (C) DPER selects privileged teacher or student evidence to rescale reward-derived token advantages while preserving their signs. Rewards combine GPRM, multi-judge rankings, and task-queue consistency. Teachers and privileged references are used only during training; deployment uses a single student.
Figure 4: Training dynamics over the first 300 steps. DPER is compared with GRPO, RLSD, and DOPD in terms of (a) actor entropy, (b) response length in tokens, and (c) maximum reward. Dark curves show smoothed trends; faint traces show step-to-step fluctuations.
(a) Benchmark-level scores ( ↑ )
Dataset
Qwen3.5-35B-A3B
Qwen3.5-397B-A17B
JoyInside-VLM-Agent
MSAgent
88.46
92.41
90.39
BFCL v4
67.30
72.90
77.44
ToolBench
18.30
21.00
23.50
ToolACE
50.92
52.27
51.53
Table 1: Tool-use performance on public benchmarks and proprietary evaluation sets. Boldface indicates the best result for each dataset and metric.
Figure 5: Visual search before greeting. The requested person is outside the quadruped’s camera view in steps 1–3, identified by clothing and laptop attributes in step 4, and seen at close range in steps 5–6. Frames follow the interaction from left to right; bounding boxes show object detections.
Figure 6: Visual search after a failed approach. The target is visible in step 1 but is not reacquired during the quadruped’s subsequent rotational search in steps 2–7. Frames follow the interaction from left to right; bounding boxes show object detections.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Table 2: Illustrative TaskQueue inverse-rewriting prompt for JoyAI-VL. The placeholder {task_queue} is replaced with the current queue.
Table 3: Product developer prompt template with configurable role, interaction interface, capabilities, and execution policies.
Table 4: Developer-role serialization. Top: the exact branch from the chat template. Bottom: a representative input with add_generation_prompt enabled and thinking enabled. Braces denote substituted content; optional tool definitions, visual-token expansion, and interaction history are omitted.
Table 5: Illustrative ReAct assistant output. The shared prefix is followed by one of the two continuations. Braced entries are placeholders; the task block is included when required by the interaction protocol.
Can a multimodal language model learn to manipulate physical objects by reasoning about its own failures-without gradient updates, demonstrations, or reward engineering? We argue the answer is yes, under conditions we characterise precisely. We present Act-Observe-Rewrite (AOR), a framework in which an LLM agent improves a robot manipulation policy by synthesising entirely new executable Python controller code between trials, guided by visual observations and structured episode outcomes. Unlike prior work that grounds LLMs in pre-defined skill libraries or uses code generation for one-shot plan synthesis, AOR makes the full low-level motor control implementation the unit of LLM reasoning, enabling the agent to change not just what the robot does, but how it does it. The central claim is that interpretable code as the policy representation creates a qualitatively different kind of in-context learning from opaque neural policies: the agent can diagnose systematic failures and rewrite their causes. We validate this across three robosuite manipulation tasks and report promising results, with the agent achieving high success rates without demonstrations, reward engineering, or gradient updates.
Physical AI requires models to ground visual and linguistic understanding in real-world environments while accounting for environmental constraints and execution feedback. We introduce MachEmbodied-VLM (ME-VLM), a unified vision-language model with two variants, 4B and 35B-A3B, that brings together embodied cognition and multimodal agent capabilities. Our work emphasizes physical perception and spatiotemporal reasoning, together with planning, interaction, and outcome assessment in both digital and physical environments. We construct training data spanning embodied and multimodal agent tasks, including execution observations and feedback to support outcome assessment and decision refinement. The training pipeline comprises embodied capability injection, separate reinforcement learning of embodied and multimodal-agent experts, and multi-teacher on-policy distillation that consolidates their complementary capabilities into a single model. Experiments show competitive performance on both embodied and agent benchmarks, as well as on autonomous-driving and embodied-navigation tasks. For edge deployment, visual token compression, W4A8 quantization, and hardware-software co-optimization enable on-device inference of the 4B variant on the M100, reducing prefill latency from 400 ms to 188 ms. Project Page: https://machembodied.com/ME-Brain/ME-VLM.html Code Repository: https://github.com/MachEmbodied/ME-VLM
Building capable embodied agents requires not only multimodal perception and understanding, but also agentic capabilities for reasoning about actions, adapting to evolving situations, and interacting with the physical world. In this report, we introduce Hy-Embodied-VLM-1.0, an efficient and powerful embodied foundation model specifically designed for embodied agents operating in the physical world. To cultivate such capabilities from the pre-training stage onward, we define an action-centric capability taxonomy comprising three progressive dimensions: Action-Relevant State Understanding, Action-Transition Reasoning, and Sequential and Adaptive Reasoning. Guided by this taxonomy, we develop a systematic data pipeline and curate data mixtures spanning both pre-training and post-training. To deliver strong physical-world understanding and interaction capabilities while supporting latency-sensitive deployment, we build our model on the Hy3-A3B language backbone and the Hy-ViT2 vision encoder. Its efficient Mixture-of-Experts architecture combines strong model capacity with high inference efficiency. We evaluate Hy-Embodied-VLM-1.0 on a comprehensive suite of 38 benchmarks covering embodied perception, physical-world understanding, and embodied reasoning. The model achieves the best performance among similarly sized models on 19 of the 38 benchmarks and substantially outperforms strong competitors, including Qwen3.6-A3B and Cosmos 3. Compared with the previous-generation Hy-Embodied-0.5 MoT-2B, Hy-Embodied-VLM-1.0 improves average performance by 8.4%. Despite activating only 3B parameters, it achieves performance close to that of the previous-generation model with 32B activated parameters. Beyond static benchmark evaluation, Hy-Embodied-VLM-1.0 also demonstrates strong performance on embodied agentic tasks requiring multi-turn interaction and long-horizon reasoning.
Ziyi Wang, Xumin Yu, Yongming Rao +19
Tencent Robotics X Hy Vision Team Futian Laboratory