UNITE: Consolidating Embodied VLM Specialists via Shared-Anchor One-Bit Factorization
Authors: Tao Tao, Anke Tang, Enneng Yang, Zhi Hou, Xiaogang Wang, Dacheng Tao
Organizations: Sichuan University, Chengdu, China · ACE Robotics, Shanghai, China · Nanyang Technological University, Singapore · Shenzhen Campus of Sun Yat-sen University, Shenzhen 518107, China
Embodied capabilities such as visual grounding, spatial reasoning, navigation, and driving understanding are often developed in separate vision-language specialists fine-tuned from a common pretrained model.Deploying the specialists separately multiplies storage, whereas merging them into one dense model forces conflicting task updates to share parameters and loses part of each specialist's expertise.We introduce UNITE, a framework that consolidates such specialists into a compact shared-backbone mixture of experts. UNITE keeps the feed-forward block of each specialist as a routable expert and represents the expert bank through a shared-anchor one-bit factorization: a shared floating-point anchor plus expert-specific, row-scaled binary deviations.The anchor is optimized jointly with the scales and signs through closed-form alternating updates, using only source weights and no calibration data.Router adaptation then adjusts token-level expert selection with the backbone and experts frozen, updating fewer than 0.6M parameters (<0.01% of the 8.8B active parameters). Built from four Qwen3-VL-8B specialists, the complete UNITE model occupies 20.3 GB, 1.16× the storage of one dense specialist and 59.6% below the uncompressed MoE. Across 22 benchmarks and R2R Val-Unseen, UNITE matches six static merging methods in mean retention of best-specialist performance, leads all of them on Scan2Cap, ScanQA, and all four navigation metrics, and surpasses every source specialist on all five 3D scene understanding benchmarks (up to +19.1 on Scan2Cap) and on R2R navigation. Relative to the uncompressed MoE, factorization is nearly lossless on 21 of 22 benchmarks; the exception is VSI-Bench, where frame-count experiments suggest impaired use of cross-frame evidence.UNITE thus consolidates complementary embodied capabilities near single-model storage.
Figures & tables
Fig. 1: Storage versus capability retention for four Qwen3-VL-8B specialists. Keeping them separate (gray) costs 70.1 GB; static merging (violet, Table I ) cuts this to 17.5 GB, but every merging method then falls below the navigation specialist’s 57.5% R2R Val-Unseen success rate, reaching at best 55.2% (Table III ). UNITE (blue) needs only 20.3 GB, 1.16× the single-specialist footprint, and surpasses the specialist at 59.6% success rate.
Fig. 2: Consolidating embodied specialists with UNITE. Grounding, spatial reasoning, navigation, and driving understanding specialists from a common pretrained model contribute separate feed-forward experts to a shared-backbone MoE. Jointly fitted anchors and one-bit deviations compress the expert bank, bringing total model storage to 1.16× that of one dense specialist; supervised router adaptation adjusts token-level expert selection.
Fig. 3: Overview of UNITE. (a) The non-FFN parameters of the dense specialists are merged into a shared backbone, while each specialist FFN is kept as an expert. (b) Alternating updates jointly fit the shared anchor C , row scales αe , and binary signs Se for each layer and weight group. (c) In the resulting MoE layer, every expert is represented by its one-bit factors (αe,Se) on top of the shared anchor C . The trainable/frozen markers refer to router adaptation, which trains only the routers while the backbone and the compressed experts stay frozen.
Fig. 4: Joint anchor fitting versus a fixed anchor. Summed reconstruction error ∑e∥We−We∥F2 as a function of the alternating iteration T , normalized per layer to its value at T=0 (fixed mean anchor with a single sign-and-scale fit). Lines are medians across the 36 decoder layers of the Qwen3-VL-8B expert bank; bands span the 10th to 90th percentile. The last point ( T=10 ) is measured after rounding the anchor and row scales to BF16 and refitting the signs and scales, i.e., the archive used at inference; the rounding costs less than one percentage point.
Fig. 5: Navigation in R2R Val-Unseen environments. Each row shows an instruction excerpt and four snapshots pairing egocentric observations with trajectory maps and action annotations. Green and blue indicate the ground-truth and UNITE trajectories, respectively. Tables III and V report navigation results.
Fig. 6: Score change contributed by each consolidation stage (top- 1 routing throughout). Left: one-bit factorization of the expert bank, reducing total model weight storage from 50.2 to 20.3 GB. Right: router adaptation at constant storage. Bars beyond ±5 points are truncated and labeled with their value. ShareRobot (Traj., on a different scale) improves from 0.34 to 0.26 and 0.25 . Full numbers are in Tables IV and V .
Fig. 7: VSI-Bench score as a function of the maximum number of sampled video frames for the uncompressed and the one-bit expert bank (same backbone, top- 1 routing, no router adaptation; all 5,130 questions). The two models are within 1.2 points at four frames. The uncompressed MoE gains 17.0 points from 4 to 64 frames, whereas the one-bit MoE stays within a two-point band, indicating limited benefit from additional frames after factorization. The main tables use 32 frames.
Fig. 8: Top- k routing before and after router adaptation. The radius is the direction-adjusted percentage change of top- k relative to top- 1 at the same stage, so the bold circle marks top- 1 and points outside it indicate improvement for every metric, including ShareRobot (Traj.) and R2R NE. The radial axis spans −6% to +5% ; NuScenesQA before router adaptation ( +6.1% and +11.8% ) exceeds this range and is clipped at the outer edge.