Robust Embodied Perception in Dynamic Environments via Disentangled Weight Fusion

Juncen Guo; Xiaoguang Zhu; Jingyi Wu; Jingyu Zhang; Jingnan Cai; Zhenghao Niu; Liang Song

cs.CVApr 2, 2026

Robust Embodied Perception in Dynamic Environments via Disentangled Weight Fusion

Authors: Juncen Guo, Xiaoguang Zhu, Jingyi Wu, Jingyu Zhang, Jingnan Cai, Zhenghao Niu, Liang Song

Organizations: Fudan University · University of California, Davis

Abstract

Embodied perception systems face severe challenges of dynamic environment distribution drift when they continuously interact in open physical spaces. However, the existing domain incremental awareness methods often rely on the domain id obtained in advance during the testing phase, which limits their practicability in unknown interaction scenarios. At the same time, the model often overfits to the context-specific perceptual noise, which leads to insufficient generalization ability and catastrophic forgetting. To address these limitations, we propose a domain-id and exemplar-free incremental learning framework for embodied multimedia systems, which aims to achieve robust continuous environment adaptation. This method designs a disentangled representation mechanism to remove non-essential environmental style interference, and guide the model to focus on extracting semantic intrinsic features shared across scenes, thereby eliminating perceptual uncertainty and improving generalization. We further use the weight fusion strategy to dynamically integrate the old and new environment knowledge in the parameter space, so as to ensure that the model adapts to the new distribution without storing historical data and maximally retains the discrimination ability of the old environment. Extensive experiments on multiple standard benchmark datasets show that the proposed method significantly reduces catastrophic forgetting in a completely exemplar-free and domain-id free setting, and its accuracy is better than the existing state-of-the-art methods.

Figures & tables

Fig. 1 : The performance of method S-iPrompts [ 18 ] under different experimental conditions. Left : the test performance of the model on domain S2 after training only on S2, and after training on both S1 and S2 (covering three datasets). Right : the model undergoes training with S1-S3 and is then tested in the subsequent unseen domain with zero samples on the CDDB [ 7 ] dataset, the model undergoes training with S1-S4 and is then tested in the subsequent unseen domain on the CORe50 [ 10 ] dataset.

Fig. 2 : The framework of disentangled representation and weight fusion for DIL.

Method	Buffer	Avg	Last
LRCIL [ 12 ]	50/class	74.01	73.42
iCaRL [ 14 ]		73.98	71.86
LUCIR [ 5 ]		80.77	78.25
LRCIL [ 12 ]	100/class	76.39	75.99
iCaRL [ 14 ]		79.76	78.51
LUCIR [ 5 ]		82.53	80.68

TABLE I : Results on CDDB.

Method	Buffer	Avg	Last
ER [ 2 ]	50/class	80.10	82.06
L2P [ 20 ]		81.07	82.72
BiC [ 5 ]		79.28	80.97
DyTox [ 3 ]		79.21	81.23
DER++ [ 1 ]		79.70	82.01
SimpleCIL [ 24 ]	0/class	70.92	74.80

TABLE II : Results on CoRe50.

Method	Buffer	Avg	Last
DyTox [ 3 ]	50/class	62.94	60.18
SimpleCIL [ 24 ]	0/class	42.95	44.08
L2P [ 20 ]		50.45	48.72
DualPrompt [ 19 ]		52.28	50.46
CODA-Prompt [ 15 ]		59.85	59.99
S-iPrompt [ 18 ]		61.16	60.46

TABLE III : Results on DomainNet.

Disen	Anti-for	Dom-inv	CDDB	CoRe50	DomainNet
✗	✗	✗	81.21	86.13	60.34
✓	✗	✗	84.53	87.86	68.59
✓	✓	✗	85.12	91.08	69.92
✓	✓	✓	86.35	92.36	71.12

TABLE IV : Ablation study of different components.

Panel of Figure 7 — Fig. 3 : The sensitivity analysis on $q$ and $\lambda$ with CDDB

Explore similar work

Feb 27, 2026

See, Act, Adapt: Active Perception for Unsupervised Cross-Domain Visual Adaptation via Personalized VLM-Guided Agent

Pre-trained perception models excel in generic image domains but degrade significantly in novel environments like indoor scenes. The conventional remedy is fine-tuning on downstream data which incurs catastrophic forgetting of prior knowledge and demands costly, scene-specific annotations. We propose a paradigm shift through Sea

^2

(See, Act, Adapt): rather than adapting the perception modules themselves, we adapt how they are deployed through an intelligent pose-control agent. Sea

^2

keeps all perception modules frozen, requiring no downstream labels during training, and uses only scalar perceptual feedback to navigate the agent toward informative viewpoints. Specially, we transform a vision-language model (VLM) into a low-level pose controller through a two-stage training pipeline: first fine-tuning it on rule-based exploration trajectories that systematically probe indoor scenes, and then refining the policy via unsupervised reinforcement learning that constructs rewards from the perception module's outputs and confidence. Unlike prior active perception methods that couple exploration with specific models or collect data for retraining them, Sea

^2

directly leverages off-the-shelf perception models for various tasks without the need for retraining. We conducted experiments on three visual perception tasks, including visual grounding, segmentation and 3D box estimation, with performance improvements of 13.54%, 15.92% and 27.68% respectively on dataset ReplicaCAD.

Tianci Tang, Tielong Cai, Hongwei Wang +1

Zhejiang University, Hangzhou, China.

Active Perception Unsupervised Domain Adaptation RL for VLMs

Ask PDF

Apr 20, 2026

XEmbodied: A Foundation Model with Enhanced Geometric and Physical Cues for Large-Scale Embodied Environments

Vision-Language-Action (VLA) models drive next-generation autonomous systems, but training them requires scalable, high-quality annotations from complex environments. Current cloud pipelines rely on generic vision-language models (VLMs) that lack geometric reasoning and domain semantics due to their 2D image-text pretraining. To address this mismatch, we propose XEmbodied, a cloud-side foundation model that endows VLMs with intrinsic 3D geometric awareness and interaction with physical cues (e.g., occupancy grids, 3D boxes). Instead of treating geometry as auxiliary input, XEmbodied integrates geometric representations via a structured 3D Adapter and distills physical signals into context tokens using an Efficient Image-Embodied Adapter. Through progressive domain curriculum and reinforcement learning post-training, XEmbodied preserves general capabilities while demonstrating robust performance across 18 public benchmarks. It significantly improves spatial reasoning, traffic semantics, embodied affordance, and out-of-distribution generalization for large-scale scenario mining and embodied VQA.

Kangan Qian, ChuChu Xie, Yang Zhong +13

Tsinghua University · Automotive and Robotics, Xiaomi Corporation · National University of Singapore +2

3D Spatial Reasoning Embodied QA VLM Reasoning 3D VLMs

Ask PDF

Jun 11, 2026

EmbodiSteer: Steering Embodiment-Agnostic Visuomotor Policies with Joint-Space Guidance for Zero-Shot Cross-Embodiment Deployment

Scalable robot imitation learning relies on large-scale heterogeneous data from diverse robots or body-free data, making Cartesian end-effector actions a key interface for embodiment-agnostic policy learning. However, end-effector-only abstraction leaves Cartesian policies unaware of the deployed robot body, making them brittle under robot-specific constraints such as whole-body collision avoidance. To overcome this limitation, we present EmbodiSteer, a training-free framework that steers embodiment-agnostic visuomotor policies toward zero-shot, embodiment-aware deployment. EmbodiSteer keeps policy learning in Cartesian space while efficiently lifting inference-time diffusion sampling into the target robot's joint space via forward kinematics and Jacobian-based updates. With whole-body collision-aware guidance over joint trajectories after each denoising step, the arm can be steered away from collisions while preserving learned end-effector behavior. Across 9 simulated robots, EmbodiSteer reduces collision rate by 46.1 percentage points and increases task success by 28.5 percentage points over Cartesian-only execution; on two physical robots in highly constrained scenarios, the respective improvements are 90.0 and 36.7 percentage points. Our project page is at https://frankwang67.github.io/EmbodiSteer-Page.

Shihefeng Wang, Kangchen Lv, Mingrui Yu +1

Department of Automation, Tsinghua University · Beijing Key Laboratory of Embodied Intelligence Systems · Institute for Embodied Intelligence and Robotics, Tsinghua University

Robot Skill Learning Generalization in Robotic Manipulation Visuomotor Policy Learning Diffusion Policy Robot Imitation Learning Cross-Embodiment Transfer+1

Ask PDF