UNITE: Consolidating Embodied VLM Specialists via Shared-Anchor One-Bit Factorization
Authors: Tao Tao, Anke Tang, Enneng Yang, Zhi Hou, Xiaogang Wang, Dacheng Tao
Organizations: Sichuan University, Chengdu, China · ACE Robotics, Shanghai, China · Nanyang Technological University, Singapore · Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China
Embodied capabilities such as visual grounding, spatial reasoning, navigation, and driving understanding are often developed in separate vision-language specialists fine-tuned from a common pretrained model.Deploying the specialists separately multiplies storage, whereas merging them into one dense model forces conflicting task updates to share parameters and loses part of each specialist's expertise.We introduce UNITE, a framework that consolidates such specialists into a compact shared-backbone mixture of experts. UNITE keeps the feed-forward block of each specialist as a routable expert and represents the expert bank through a shared-anchor one-bit factorization: a shared floating-point anchor plus expert-specific, row-scaled binary deviations.The anchor is optimized jointly with the scales and signs through closed-form alternating updates, using only source weights and no calibration data.Router adaptation then adjusts token-level expert selection with the backbone and experts frozen, updating fewer than 0.6M parameters (<0.01% of the 8.8B active parameters). Built from four Qwen3-VL-8B specialists, the complete UNITE model occupies 20.3 GB, 1.16× the storage of one dense specialist and 59.6% below the uncompressed MoE. Across 22 benchmarks and R2R Val-Unseen, UNITE matches six static merging methods in mean retention of best-specialist performance, leads all of them on Scan2Cap, ScanQA, and all four navigation metrics, and surpasses every source specialist on all five 3D scene understanding benchmarks (up to +19.1 on Scan2Cap) and on R2R navigation. Relative to the uncompressed MoE, factorization is nearly lossless on 21 of 22 benchmarks; the exception is VSI-Bench, where frame-count experiments suggest impaired use of cross-frame evidence.UNITE thus consolidates complementary embodied capabilities near single-model storage.
Figures & tables
Fig. 1: Storage versus capability retention for four Qwen3-VL-8B specialists. Keeping them separate (gray) costs 70.1 GB; static merging (violet, Table I ) cuts this to 17.5 GB, but every merging method then falls below the navigation specialist’s 57.5% R2R Val-Unseen success rate, reaching at best 55.2% (Table III ). UNITE (blue) needs only 20.3 GB, 1.16× the single-specialist footprint, and surpasses the specialist at 59.6% success rate.
Fig. 2: Consolidating embodied specialists with UNITE. Grounding, spatial reasoning, navigation, and driving understanding specialists from a common pretrained model contribute separate feed-forward experts to a shared-backbone MoE. Jointly fitted anchors and one-bit deviations compress the expert bank, bringing total model storage to 1.16× that of one dense specialist; supervised router adaptation adjusts token-level expert selection.
Fig. 3: Overview of UNITE. (a) The non-FFN parameters of the dense specialists are merged into a shared backbone, while each specialist FFN is kept as an expert. (b) Alternating updates jointly fit the shared anchor C , row scales αe , and binary signs Se for each layer and weight group. (c) In the resulting MoE layer, every expert is represented by its one-bit factors (αe,Se) on top of the shared anchor C . The trainable/frozen markers refer to router adaptation, which trains only the routers while the backbone and the compressed experts stay frozen.
Fig. 4: Joint anchor fitting versus a fixed anchor. Summed reconstruction error ∑e∥We−We∥F2 as a function of the alternating iteration T , normalized per layer to its value at T=0 (fixed mean anchor with a single sign-and-scale fit). Lines are medians across the 36 decoder layers of the Qwen3-VL-8B expert bank; bands span the 10th to 90th percentile. The last point ( T=10 ) is measured after rounding the anchor and row scales to BF16 and refitting the signs and scales, i.e., the archive used at inference; the rounding costs less than one percentage point.
Fig. 5: Navigation in R2R Val-Unseen environments. Each row shows an instruction excerpt and four snapshots pairing egocentric observations with trajectory maps and action annotations. Green and blue indicate the ground-truth and UNITE trajectories, respectively. Tables III and V report navigation results.
Fig. 6: Score change contributed by each consolidation stage (top- 1 routing throughout). Left: one-bit factorization of the expert bank, reducing total model weight storage from 50.2 to 20.3 GB. Right: router adaptation at constant storage. Bars beyond ±5 points are truncated and labeled with their value. ShareRobot (Traj., on a different scale) improves from 0.34 to 0.26 and 0.25 . Full numbers are in Tables IV and V .
Fig. 7: VSI-Bench score as a function of the maximum number of sampled video frames for the uncompressed and the one-bit expert bank (same backbone, top- 1 routing, no router adaptation; all 5,130 questions). The two models are within 1.2 points at four frames. The uncompressed MoE gains 17.0 points from 4 to 64 frames, whereas the one-bit MoE stays within a two-point band, indicating limited benefit from additional frames after factorization. The main tables use 32 frames.
Fig. 8: Top- k routing before and after router adaptation. The radius is the direction-adjusted percentage change of top- k relative to top- 1 at the same stage, so the bold circle marks top- 1 and points outside it indicate improvement for every metric, including ShareRobot (Traj.) and R2R NE. The radial axis spans −6% to +5% ; NuScenesQA before router adaptation ( +6.1% and +11.8% ) exceeds this range and is clipped at the outer edge.
Embodied and edge systems increasingly rely on vision-language models (VLMs) to support heterogeneous skills, yet deploying a separate task adapter for each skill while retaining a high-precision backbone creates substantial memory and management costs. We present UniCompress, a deployment-oriented framework that converts multi-task expertise into one compact, adapter-free model. Starting from a consolidated BF16 Qwen3-VL-4B teacher, UniCompress constructs an asymmetric INT2 student using GPTQ initialization and teacher-guided quantization recovery. The approach jointly quantizes and recovers the projection weights in the vision encoder and language backbone, enabling aggressive compression of the complete vision-language model while preserving its multi-task behavior. We evaluate UniCompress on four heterogeneous embodied tasks: posture recognition, household-safety understanding, multimodal dialogue routing, and mobile-GUI action prediction. UniCompress reduces the approximate weight storage from 8.9 GB in BF16 to 2.1 GB in INT2, corresponding to a reduction of approximately 76%. It improves the INT2 macro task score by 34.3 percentage points over GPTQ-INT2 while remaining within 0.9 points of the consolidated BF16 teacher. These results demonstrate the potential of teacher-guided joint vision-language INT2 recovery for deploying a single multi-skill VLM on resource-constrained embodied platforms.
We present Pelican-Unify 1.0, the first embodied foundation model trained according to the principle of unification. Pelican-Unify 1.0 uses a single VLM as a unified understanding module, mapping scenes, instructions, visual contexts, and action histories into a shared semantic space. The same VLM also serves as a unified reasoning module, autoregressively producing task-, action-, and future-oriented chains of thought in a single forward pass and projecting the final hidden state into a dense latent variable. A Unified Future Generator (UFG) then conditions on this latent variable and jointly generates future videos and future actions through two modality-specific output heads within the same denoising process. The language, video, and action losses are all backpropagated into the shared representation, enabling the model to jointly optimize understanding, reasoning, imagination, and action during training, rather than training three isolated expert systems. Experiments demonstrate that unification does not imply compromise. With a single checkpoint, Pelican-Unify 1.0 achieves strong performance across all three capabilities: 64.7 on eight VLM benchmarks, the best among comparable-scale models; 66.03 on WorldArena, ranking first; and 93.5 on RoboTwin, the second-best average among compared action methods. These results show that the unified paradigm succeeds in preserving specialist strength while bringing understanding, reasoning, imagination, and action into one model.
Yi Zhang, Yinda Chen, Che Liu +26
Beijing Innovation Center of Humanoid Robotics (X-Humanoid) WFM System Group
Co-training offers a straightforward way to build a multi-task vision-language-action (VLA) policy, but can fall short of the performance achieved by training each task independently. The challenge is to retain these task-specific gains in a multi-task policy without joint post-training. Combining independently trained experts through model merging is a natural approach, yet strong individual experts do not necessarily yield a strong merged policy. We identify one source of this incompatibility: task-specific changes to the action interface, comprising action normalization and the action encoder and decoder. We propose PolicyWeave, combining merge-compatible post-training with context-guided sparse merging. During post-training, all experts retain the common base policy's action interface, while task adaptation is restricted to LoRA updates in the hidden layers of the action model. This makes the experts more compatible with existing model merging methods. However, merging all experts can still introduce interference from unrelated tasks at deployment. PolicyWeave scores each expert's LoRA updates using the initial visual-language context, determines the expert set through leave-one-layer-out ranking stability, and forms a sparse weighted merge of the selected updates that remains fixed for current task. We evaluate PolicyWeave with GR00T N1.5 on 18 RoboCasa365 tasks, using only 10% of the target-task demonstrations for supervised fine-tuning (SFT). Preserving the shared action interface raises the average success rate across four static merging methods from 17.0% to 52.8%. PolicyWeave achieves 64.7% success with these SFT experts and 74.1% after task-specific reinforcement learning (RL), compared with 60.7% for joint RL. Further evaluations on LIBERO-10 and an AgileX Piper arm support the deployment of independently learned skills in long-horizon and real-world manipulation.