Action Chunking Proximal Policy Optimization with Feedback Correction

Sanghyun Hahn; Jonghyun Choi

cs.LGSep 28, 2026

Action Chunking Proximal Policy Optimization with Feedback Correction

Authors: Sanghyun Hahn, Jonghyun Choi

Organizations: Cornell University · Seoul National University

Abstract

Action chunking provides temporal abstraction in reinforcement learning by selecting short action sequences instead of individual actions, but many existing approaches face two limitations in high-dimensional robotic control. First, many rely on value functions over action chunks, which can be difficult to learn as action dimensionality and chunk length grow. Second, executing chunks open-loop removes within-chunk feedback, limiting reactivity in contact-rich tasks. We present Action Chunking PPO (ACPPO), a PPO extension that uses a chunked actor while retaining a standard state-value critic, thereby avoiding chunked Q-functions. We further propose ACPPO-Corr, which augments the chunk planner with a stepwise feedback corrector that adjusts planned actions online within each chunk. Across 25 simulated robotics tasks from IsaacGym and Bi-DexHands, spanning locomotion, arm manipulation, and dexterous hand-object interaction, ACPPO-Corr achieves the strongest aggregate performance among evaluated methods and performs best on both decision-frequency-sensitive and decision-frequency-neutral task subsets. Ablations show that moderate chunk lengths work best and that corrector regularization is important for balancing chunk-level planning with local feedback. These results suggest that action chunking can be effective in online PPO when chunk-level planning is paired with closed-loop correction. The code is available at: https://github.com/hshhahn/ACPPO.

Figures & tables

Figure 1 : Stepwise comparison between PPO, ACPPO, and ACPPO-Corr. PPO runs the policy closed-loop. ACPPO runs the policy open-loop. ACPPO-Corr corrects the open-loop policy with a closed-loop feedback corrector. In ACPPO-Corr, the open-loop policy and the closed-loop corrector are jointly trained.

Panel of Figure 2 — Figure 2 : Benchmark results. (a) Normalized IQM Scores over training across all 25 tasks. Shaded regions show pointwise 95% seed-bootstrap intervals with tasks fixed. ACPPO-Corr achieves best aggregated performance, improving over PPO by $30.4\%$ in final normalized IQM. (b) Final relative IQM with 95% seed-bootstrap intervals with tasks fixed on the frequency-sensitive and frequency-neutral subsets, computed as the ratio to PPO’s IQM on the same task set.

Figure 3 : Rollout snapshots of PPO, ACPPO, and ACPPO-Corr on AllegroHand . ACPPO-Corr reaches the target orientation the fastest, where ACPPO drops the cube and PPO requires additional 128 environment interactions to reach the target.

Panel of Figure 4 — Figure 4 : Ablations on chunked advantage, corrector regularization, and chunk length. (a) Chunked GAE and corrector regularization improve both performance and stability. (b) $h=1$ removes multi-step chunking while retaining the ACPPO-Corr pipeline. Moderate chunk lengths perform best, with $h=4$ giving the strongest aggregate performance.

Figure 5 : Sensitivity to the corrector regularizer $\lambda_{r}$ on three representative tasks. Different tasks prefer different regularization strengths: $\lambda_{r}$ is a relevant method-specific hyperparameter.

Table 6

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Tasks	#	W/T/L
Neutral	FrankaCabinet W , Ingenuity T , ShadowHandDoorOpenInward W , ShadowHandDoorCloseInward T , ShadowHandDoorOpenOutward T , ShadowHandScissors W , QuadCopter T , ShadowHandGraspAndPlace W , ShadowHandOver T , ShadowHandBlockStack W	10	5/5/0
Sensitive	ShadowHandSpin W , ShadowHandBottleCap L , Humanoid W , Anymal T , ShadowHand W , AllegroHand W , Ant W , ShadowHandUpsideDown L , ShadowHandCatchOver2Underarm W , ShadowHandCatchAbreast W , FrankaCubeStack T	11	7/2/2
Excluded	ShadowHandLiftUnderarm W , ShadowHandDoorCloseOutward L , ShadowHandKettle L , ShadowHandPushBlock T	4	1/1/2
All tasks	–	25	13/8/4

Appendix

Table 3: Task assignment for the frequency-sensitivity split and per-task win/tie/loss status of ACPPO-Corr. Superscripts denote ACPPO-Corr’s status against the strongest competing method on each task: W = win, T = tie, L = loss.

Algorithm	Network components
PPO	1 full-size actor + 1 full-size value critic
ACPPO	1 full-size chunked actor + 1 full-size value critic
ACPPO-Corr	1 chunk planner + 1 corrector + 1 full-size value critic
SAC	1 actor + 2 Q-functions + 2 target Q-functions
QC-FQL	1 chunked actor + 2 Q-functions + 2 target Q-functions + 1 flow network

Appendix

Table 4 : Network components used per algorithm. Halved MLP configuration is used for methods with multiple actor or critic branches.

Algorithm	Steps/sec	Parameters
PPO	47,894	1,463,785
ACPPO	45,579	1,484,065
ACPPO-Corr	41,714	1,557,258
SAC	39,049	2,024,428
QC-FQL	34,595	3,287,492

Appendix

Table 5 : Training throughput and model size. Steps/sec includes rollout and optimization. Parameter counts include policy/value networks and, for off-policy methods, target critic networks used during training.

Figure 6 : Full training curves on IsaacGym and Bi-DexHands.

Task	$\lambda_{r}$ $(h=1)$	$\lambda_{r}$ $(h=2)$	$\lambda_{r}$ $(h=4)$	$\lambda_{r}$ $(h=8)$	QC-FQL $\alpha$	$R_{\mathrm{low}}$	$R_{\mathrm{high}}$
FrankaCabinet	0.1	0.1	0.1	0.1	3	0	1
ShadowHandDoorOpenInward	0.1	0.3	0.1	0.1	1	0	1
ShadowHandDoorCloseInward	0.3	0.1	0.1	0.1	1	0	1
ShadowHandDoorOpenOutward	0.3	0.3	0.3	0.3	1	0	1
ShadowHandScissors	0.3	0.3	0.3	0.3	1	0	1
Ingenuity	0.1	0.1	0.1	0.1	1	0	14000

Appendix

Table 6 : Per-task selected hyperparameters and normalization bounds.

\lambda_{r}

denotes the corrector regularization coefficient for ACPPO-Corr at each chunk length,

\alpha

denotes the behavior regularization coefficient for QC-FQL, and

R_{\mathrm{low}},R_{\mathrm{high}}

are the task-specific bounds used in normalized IQM computation.

Task	Physics (Hz)	Action repeat	Control (Hz)	$h=4$ (s)
AllegroHand	120	4	30	0.133
Ant	120	2	60	0.067
Anymal	100	2	50	0.080
FrankaCabinet	60	1	60	0.067
FrankaCubeStack	120	2	60	0.067
Humanoid	120	2	60	0.067

Appendix

Table 7 : Simulation and control timescales. The ShadowHand row includes all ShadowHand-prefixed tasks.

Method	Task	$h=4$	$h=8$	$h=16$	$h=32$
ACPPO-Corr	FrankaCubeStack	2.21	2.17	1.72	1.09
	Humanoid	1.11	1.07	0.97	0.65
	Ingenuity	0.99	1.00	0.99	–
	FrankaCabinet	1.56	1.39	1.48	–
	Ant	1.06	0.97	0.88	–
ACPPO	FrankaCubeStack	2.11	1.90	1.37	0.00

Appendix

Table 8 : Per-task score ratios to PPO at longer chunk lengths. A dash denotes an unevaluated setting due to the rollout horizon.

Method	Chunk length	Clip fraction
PPO	–	0.365
ACPPO-Corr	2	0.364
ACPPO-Corr	4	0.369
ACPPO-Corr	8	0.408

Appendix

Table 9 : Chunk-level clipping diagnostics.

Explore similar work

Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control?
Aug 3, 2026Filippo Lazzati, Kyle Stachowicz, William Chen +3Action Chunks Robot Policies
Politecnico di Milano · UC Berkeley
Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning
May 7, 2026Nandiraju Gireesh, Yuanliang Ju, He WangAction Chunks Chunk
Peking University · Galbot · University of Toronto
PACE: Phase-Aware Chunk Execution for Robot Policies with Action Chunking
May 30, 2026Junnan Nie, Jiayi Li, Chenghao Liu +5Action Chunks Robot Policies
Peking University · Peking University. · JD Explore Academy. +1

Aug 3, 2026, cs.ROJ/K move · Enter open · S save

Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control?

Filippo Lazzati, Kyle Stachowicz, William Chen, Alberto Maria Metelli+2

Politecnico di Milano · UC Berkeley

Action Chunks Robot Policies Behavior Cloning Ensemble Expressivity

Open paper AskPDF arXiv

Action chunking---predicting and executing multiple actions instead of a single action---has proven to be a critical component for learning effective robotic control policies. However, our precise understanding of why action chunking improves performance has remained limited. In this work we seek to close this gap. Through rigorous experimental evaluations in both simulated and real-world settings, we show that existing hypotheses for the success of action chunking---temporal consistency, horizon reduction, and representation learning---fail to explain the success of action chunking. Instead, we find that action chunking benefits from greater non-Markovian expressivity and reduced compounding error compared to Markovian policies, but, in many settings of interest, these effects can be fully captured by delayed policies, which at each step predict a single action based on the observation

k

steps in the past. We then show that there exists an additional benefit of action chunking that we refer to as implicit ensembling. In particular, by learning a diversity of temporal relationships (that is,

a_t | o_t, a_t | o_{t-1}, \ldots

), action-chunked policies exhibit behavior matching that of a model ensemble, increasing their robustness and generalization ability over policies that only learn a single temporal relationship. Building on these insights, we show that in simulated and real-world robotic control settings, we can match the performance of action chunking without action chunking---by deploying an action chunking policy as an ensemble of policies with randomized delays. Furthermore, we propose a policy class that amplifies the benefits of action chunking by explicitly instantiating an ensemble, and which we show significantly improves over the performance of action chunking in many domains.

Abstract

Explore similar work

Why Does Action Chunking Improve Behavioral Cloning Performance in Robotic Control?

Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning

PACE: Phase-Aware Chunk Execution for Robot Policies with Action Chunking