CSO: Refining Robotic Policies via Skill Distribution Alignment and Skill-Grained Optimization

Zhiyuan Xiang; Xiang Deng; Wanjie Tao; Xu Chen; Zhao Jielun; Weili Guan

cs.ROOct 28, 2026

CSO: Refining Robotic Policies via Skill Distribution Alignment and Skill-Grained Optimization

Authors: Zhiyuan Xiang, Xiang Deng, Wanjie Tao, Xu Chen, Zhao Jielun, Weili Guan

Organizations: Harbin Institute of Technology (Shenzhen) · Mashang Consumer Finance Co., Ltd. · Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China

Abstract

Discretizing continuous actions into discrete skills using methods like VQ-VAE has emerged as a powerful paradigm for robotic manipulation. However, quantization errors yield a suboptimal distribution for the skill prior, degrading its performance. While reinforcement learning offers a path for refinement, its direct application faces unstable encoder updates and a granularity dilemma in importance sampling. To address these challenges, we introduce Cascaded Skills Optimization (CSO), a two-stage post-training framework. First, Rejection-Sampling Supervised Fine-tuning aligns the model's observation-to-skill mapping with the distribution of successful online trajectories via supervised fine-tuning. Second, with the observation encoder frozen, Skills Policy Optimization computes an independent, clipped importance ratio for each complete skill, enabling stable and efficient updates. With a lightweight QueST-style policy, CSO raises overall LIBERO success from 80.1% to 97.5%, improves performance on RoboTwin2.0, and achieves 61.0% success across five physical single-arm tasks. Additional comparisons with FAST, discrete BEAST, and fixed-buffer IQL further validate the effectiveness of our two-stage post-training strategy.

Figures & tables

Figure 1: The architecture and data flow of our proposed CSO framework. Stage 1 (R-SFT) : The initial policy is executed to collect trajectories, and successful trajectories are stored in a replay buffer. The observation-to-skill policy is then fine-tuned on this curated dataset using a supervised objective; the action tokenizer and decoder remain fixed. Stage 2 (SPO) : With the encoders now frozen, the policy collects groups of trajectories. The SPO algorithm then updates the skill prior with a skill-level policy gradient based on terminal group-relative advantages.

Method	LIBERO-Object	LIBERO-Spatial	LIBERO-Goal	LIBERO-Long	Overall
Octo † [ 29 ]	$85.7\pm 0.9$	$78.9\pm 1.0$	$84.6\pm 0.9$	$51.1\pm 1.3$	$75.1\pm 0.6$
OpenVLA † [ 14 ]	$88.4\pm 0.8$	$84.7\pm 0.9$	$79.2\pm 1.0$	$53.7\pm 1.3$	$76.5\pm 0.6$
Dita † [ 12 ]	$96.3\pm 0.7$	$84.2\pm 1.0$	$85.4\pm 1.5$	$63.8\pm 0.8$	$82.4\pm 0.8$
$\pi_{0}$ [ 2 ]	$98.8\pm 0.5$	$96.8\pm 0.8$	$95.8\pm 1.0$	$85.2\pm 1.1$	$94.2\pm 0.6$
$\pi_{0}$ + FAST † [ 31 ]	$96.8\pm 0.7$	$96.4\pm 0.9$	$88.6\pm 0.8$	$60.2\pm 0.2$	$85.5$
ResNet-T [ 28 ]	$78.9\pm 1.4$	$75.7\pm 1.9$	$52.7\pm 2.4$	$45.0\pm 1.1$	$63.1$

Table 1: Success rates (%) on LIBERO. Overall is the unweighted mean of the four displayed suites. Baselines marked with † are cited from their original papers. Reported uncertainties are standard deviations over three random seeds. Methods are grouped by architectural paradigm. The best and second-best results are highlighted.

Method	Clean (%)	Randomized (%)	Average (%)
QueST [ 28 ]	$67.0\pm 0.7$	$65.3\pm 0.7$	$66.2\pm 0.7$
QueST + RIPT [ 37 ]	$76.6\pm 0.4$	$75.3\pm 0.9$	$76.0\pm 0.5$
CSO (QueST-style)	$79.6\pm 1.3$	$75.7\pm 0.9$	$77.7\pm 0.8$
CSO (Qwen3-VL-2B)	$81.2\pm 1.2$	$79.6\pm 0.6$	$80.4\pm 0.7$

Table 2: RoboTwin2.0 results. We report success rates (%) under clean and randomized evaluation settings, together with their average.

Method / Configuration	LIBERO-Long (%)	LIBERO-90 (%)
(1) IL-pretrained policy (Baseline)	69.1	88.6
(2) CSO (Ours, Full Model)	96.0	97.7
(3) w/o SPO (R-SFT only)	85.3	93.1
(4) w/o R-SFT (SPO only)	91.7	95.3
(5) w/o Encoder Freeze	89.6	94.4

Table 3: Ablation study on the core components of our CSO framework. The synergistic combination of R-SFT and SPO, together with encoder freezing, achieves the strongest performance.

Method	Update or credit scheme	SR (%)
GRPO	Token-level ratio	94.5
GSPO	Episode-level ratio	93.2
PPO	Learned value critic	94.3
SPO	Skill-level ratio	96.0

Table 4: Stage-2 optimization from the same R-SFT checkpoint on LIBERO-Long. Each method uses its best evaluated clipping range.

Figure 2: LIBERO-90 learning curves: mean reward with standard-deviation shading over three seeds. Table 4 separately evaluates LIBERO-Long.

Representation	Reconstruction MSE	IL	R-SFT	Full CSO
FSQ/QueST-style	0.042	69.1	85.3	96.0
FAST	0.018	78.2	89.5	97.2
Discrete BEAST	0.015	84.5	91.2	98.1

Table 5: Matched action representations on LIBERO-Long. Success rates (IL, R-SFT, CSO) are percentages; MSE measures reconstruction.

Method	Cube on Cube	Cube in Bowl	Press Button	Cube on Bottle	Sequential Placement	Average
VQ-BeT	45	35	25	35	5	29.0
QueST	45	50	50	50	10	41.0
QueST + RIPT	60	60	70	60	25	55.0
CSO	65	70	80	60	30	61.0

Table 6: Success rates (%) on five physical tasks, with 20 trials per task.

Figure 3: Qualitative results for the Cube in Bowl task. Top (Before CSO): The initial imitation-learned policy fails to grasp the cube due to an incorrect gripper orientation. Bottom (After CSO): The CSO-refined policy corrects the pose for a successful grasp, demonstrating improved fine-grained control.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Multitask performance on the LIBERO 90 and Metaworld ML45 benchmarks. Results show the mean and standard error across three random seeds.

Method	Stage 1 Success Rate	Stage 2 Success Rate (Conditional)	Overall Success Rate
	(Cube into Bowl)	(Bowl onto Plate \| Stage 1 Success)
VQ-BeT (IL Baseline)	20%	25%	5%
QueST (IL Baseline)	40%	25%	10%
QueST + RIPT (RL Baseline)	50%	50%	25%
CSO (Ours)	60%	50%	30%

Appendix

Table 7: Stage-wise Success Rate (%) analysis for the real-world "Sequential Placement" task.

Benchmark Suite	Initial IL Success Rate	Rollouts Needed (Avg.)
LIBERO-Long	69.1%	$\sim$ 538
LIBERO-90	88.6%	$\sim$ 305
MetaWorld ML45	85.0%	$\sim$ 341

Appendix

Table 8: Average data collection cost to fill the R-SFT buffer (200 successes).

Sample Efficiency (R-SFT)		Iterative vs. Two-Stage
# Samples	Success Rate (%)	Strategy	Success Rate (%)
50	91.5 $\pm$ 0.7	Two-Stage (Ours)	96.0 $\pm$ 0.5
100	94.2 $\pm$ 0.5	Iterative Loop	96.3 $\pm$ 0.6
200	96.0 $\pm$ 0.5	Training Cost	2.5 $\times$ Time

Appendix

Table 9: Ablations on LIBERO-Long: sample efficiency (left) and design strategy (right).

Skill Length	Success Rate (%)	Observation
4	93.5	More aggressive compression.
8 (Ours)	96.0	Best evaluated value.
16	91.6	Longer autoregressive generation.

Appendix

Table 10: Effect of Skill Sequence Length on Performance.

Action chunk $k$	IL SR (%)	CSO SR (%)	Gain
16	62.4	88.5	+26.1
32	69.1	96.0	+26.9
64	54.2	79.8	+25.6

Appendix

Table 11: Action chunk length sensitivity on LIBERO-Long at

n=8

tokens per skill. Gains are absolute percentage points over the corresponding initial policy.

Method	Clipping Range	Success Rate	Clipping Fraction
Token (GRPO)	$[0.9,1.1]$	93.8%	1.2%
	$[0.8,1.2]$	94.2%	3.1%
	$[0.8,1.28]$ (Best)	94.5%	3.5%
Trajectory (GSPO)	$[0.9997,1.0004]$	92.1%	0.8%
	$[0.997,1.004]$ (Best)	93.2%	1.5%
	$[0.97,1.04]$	86.4%	12.3%

Appendix

Table 12: Clipping range sensitivity analysis. SPO maintains high performance across ranges, while GSPO is highly sensitive to the clipping bounds.

Method	Credit Assignment	LIBERO-Long SR
SPO (Ours)	Critic-free / Shared terminal advantage	96.0%
Standard PPO	Learned Critic / Fine-grained	94.3%

Appendix

Table 13: Comparison of Critic-Free (SPO) vs. Critic-Based (PPO) optimization.

Protocol	SR (%)
Expert-only IL	69.1
Joint training from scratch	76.8
Mixed fine-tuning	80.2
Sequential R-SFT	85.3

Appendix

Table 14: Data and training-protocol comparison on LIBERO-Long. All alternatives using collected successes have access to the same success set.

Method	Data strategy	SR (%)
R-SFT only	No Stage-2 optimization	85.3
R-SFT + IQL	Fixed offline buffer	89.4
R-SFT + SPO	Adaptive online collection	96.0

Appendix

Table 15: Fixed-buffer and adaptive online refinement on LIBERO-Long, from the same R-SFT initialization and with matched Stage-2 interaction cost.

Expert demonstrations per task	Initial SR (%)	Final CSO SR (%)
10	12.0	82.4
25	38.5	91.8
50	69.1	96.0

Appendix

Table 16: Refinement of weaker initial policies on LIBERO-Long.

Perturbation	QueST	QueST + RIPT	CSO
RGB/camera perturbation	42.3	71.0	88.4
Proprioceptive noise	51.0	76.2	91.2
Action delay	38.5	65.4	84.6

Appendix

Table 17: Success rates (%) under the reported perturbation conditions. Results are point estimates for each perturbation condition.

Failure category	Trials
Unseen configurations or incorrect skill selection	14
Insufficient tokenizer precision or missing motion patterns	11
Contact, grasp, or slippage errors	8
Long-horizon subgoal errors	4
Hardware or safety stops	2
Total	39

Appendix

Table 18: Failure categories for the 39 unsuccessful physical CSO evaluation trials.

Stage or strategy	Attempts	Wall-clock hours	GPU hours
CSO R-SFT	$\sim$ 5,380	3.5	14.0
CSO SPO	$\sim$ 41,200	12.0	48.0
Complete CSO	$\sim$ 46,580	15.5	62.0
RIPT, full budget matched	$\sim$ 46,580	14.5	58.0

Appendix

Table 19: LIBERO-Long post-training cost. GPU hours are aggregated across the four A100 80GB GPUs. RIPT is separately matched to CSO’s complete attempted-rollout budget.

Stage	Attempts	Retained data	Robot h	GPU h	Recoveries
R-SFT	$\sim$ 240	100 successes (20/task)	6.5	2.0	42
SPO	$\sim$ 600	400 trajectories	15.0	5.0	85
Total	$\sim$ 840	100 successes + 400 trajectories	21.5	7.0	127

Appendix

Table 20: Physical post-training across all five tasks. Robot time includes routine between-trial scene setup. Additional recoveries count workspace-boundary and safety-stop interventions. GPU hours refer to optimization compute, separately from robot hours.

Robot Policy Learning

Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.

See all 377 papers

ARP: Enhancing Quantized Skill Abstractions via Visual Alignment and Iterative Refinement for Robotic Manipulation
Jun 21, 2026Yuntian Wang, Zesheng Jia, Yuhui Duan +5Long-Horizon Robotic Manipulation Robot Skill Learning
School of Future Science and Engineering, Soochow University · Key Laboratory of General Artificial Intelligence and Large Models in Provincial Universities, Soochow University · College of Mechanical and Electronic Engineering, China Jiliang University +2
AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization
Aug 21, 2026Huizu Lin, Chengkai Huang, Tianqi Gao +5LLM Agent Skill Learning Agent Skill Learning
University of New South Wales Sydney, Australia · University of Science and Technology of China, Hefei Comprehensive National Science Center Hefei, China
OrthoSkillVLA: Continual Skill Learning via Gradient-Informed Skill Subspace Adaptation
Aug 20, 2026Jiaqi Wang, Zhou Fang, Qiongfeng Shi +1Continual Learning for VLA Models Continual Robot Learning
School of Computer Science and Engineering, Southeast University, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China · School of Electronic Science and Engineering, Southeast University, China

Jun 21, 2026J/K move · Enter open · S save

ARP: Enhancing Quantized Skill Abstractions via Visual Alignment and Iterative Refinement for Robotic Manipulation

Yuntian Wang, Zesheng Jia, Yuhui Duan, Qibing Wang+4

School of Future Science and Engineering, Soochow University · Key Laboratory of General Artificial Intelligence and Large Models in Provincial Universities, Soochow University · College of Mechanical and Electronic Engineering, China Jiliang University+2

Long-Horizon Robotic Manipulation Robot Skill Learning Visuomotor Policy Learning Imitation Learning Interactive Imitation Learning Robot Imitation Learning

Open paper AskPDF arXiv

Learning visuomotor policies for long-horizon manipulation remains a fundamental challenge. Recent skill-based imitation learning methods based on discrete quantization have shown promising results by representing complex behaviors as temporally extended skills. However, most existing approaches primarily encode action trajectories into latent skills, yielding weak visual-semantic grounding and limiting the ability to leverage visual observations for skill selection. Moreover, discrete tokenization inevitably incurs precision errors during continuous action generation. To alleviate these issues, we propose Aligned Refinement Policy (ARP), a discrete-skill framework that couples semantic grounding with execution-level refinement. Specifically, ARP introduces (i) a visual--action alignment objective that contrastively aligns visual embeddings with pre-quantized action representations in a shared latent space while preserving a state-independent skill decoder, and (ii) a lightweight Iterative Residual Head (IRH) that performs a two-step refinement to recover fine-grained control for precise execution. Extensive experiments show that ARP achieves state-of-the-art performance on the LIBERO and Meta-World benchmarks. Moreover, real-robot experiments on the Kuavo 4 Pro humanoid platform further validate its effectiveness, yielding consistent performance gains over several baselines on two challenging manipulation tasks.

Abstract

Robot Policy Learning

Explore similar work

ARP: Enhancing Quantized Skill Abstractions via Visual Alignment and Iterative Refinement for Robotic Manipulation

AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization

OrthoSkillVLA: Continual Skill Learning via Gradient-Informed Skill Subspace Adaptation