SHORE: Stability-Hybrid On-Policy Reinforcement Learning for Online Bin Packing

Chang Shi; Ann Majewicz Fey

cs.RONov 5, 2026

SHORE: Stability-Hybrid On-Policy Reinforcement Learning for Online Bin Packing

Authors: Chang Shi, Ann Majewicz Fey

Organizations: University of Texas at Austin, Austin, TX 78712, USA

Abstract

Online 3D bin packing with a processing buffer requires choosing which box to pack, its orientation, and its placement while keeping the stack stable under unknown future arrivals. A collapse ends the episode and forfeits the remaining volume. Existing methods use geometric rules or hard stability masks, limiting how risk can influence the joint choice of box, orientation, and placement. We propose Stability-Hybrid On-Policy Reinforcement Learning (SHORE), which combines a learned stability probability of a placement with an on-policy estimate of its return conditional on stability. A 3D U-Net learns stability from executed outcomes and simulated alternatives, revealing outcomes of actions underrepresented in early rollouts. A discrete on-policy actor--critic uses stability predictions in both its action values and policy prior, with gates that limit the effect of an untrained or changing model. In deployment, SHORE does not require additional search or simulation overhead. In Isaac Gym, SHORE reaches 73% space utilization by 0.4M environment steps, while the same learner without stability modeling reaches that level at about 1M. On the expanded 60-box set, SHORE achieves 0.734 space utilization versus 0.588 for the GOPT state-of-the-art baseline, with a higher placement-failure frequency (1.03% vs. 0.63%), trading more risk for greater packed volume. The results show faster learning and higher utilization on the more demanding realistic box arrival set. Ablations suggest that simulated alternatives provide useful early supervision, while executed outcomes alone sustain later performance. We also demonstrate the method on a real robot to verify the simplicity of integrating the method to real bin packing pipelines.

Figures & tables

Fig. 1 : Illustration of the online bin packing setting with a finite processing buffer. The packing agent selects a box from the buffer, its orientation, and its placement while future arrivals remain unknown.

Fig. 2 : SHORE framework. The bin configuration $C_{t}$ and the buffer $d_{t}$ are encoded as EMS tokens and (box, orientation) item tokens, and the rule-based mask $m$ keeps the geometrically valid pairs. The actor, a Packing Transformer, scores every valid pair and samples the action $a=(u,k)$ , which Isaac Gym executes and adjudicates by Eq. ( 2 ). A second Packing Transformer, the critic, values every pair in one pass, and a discretized REPPO learner updates both from on-policy rollouts (Sec. IV-B ).

Dimension (cm)	Density (kg/m 3 )	Rigidity	Counts
6 × 6 × 4	100	0.5	10
6 × 6 × 6	100	0.5	10
6 × 6 × 6	5000	3	10
8 × 6 × 6	5000	3	10

TABLE I : Box specifications in the simulation environment.

ID	Reference (in)	Simulation (cm)	Count
B01	$8\times 8\times 6$	$4\times 4\times 3$	5
B02	$10\times 8\times 6$	$5\times 4\times 3$	5
B03	$12\times 8\times 6$	$6\times 4\times 3$	5
B04	$12\times 10\times 6$	$6\times 5\times 3$	5
B05	$12\times 12\times 8$	$6\times 6\times 4$	5
B06	$14\times 10\times 8$	$7\times 5\times 4$	5

TABLE II : Box geometries for the FBA [ 19 ] -inspired synthetic benchmark. Each geometry appears five times per episode. Density is assigned separately from geometry.

	OL Mask [ 3 ]	GOPT [ 2 ]	GOPT+	SHORE(Ours)
Space utilization( $\uparrow$ )	0.495	0.661	0.683	0.657
Mean Episode Length( $\uparrow$ )	28.68	38.25	39.53	38.18
Unstable Rate( $\downarrow$ )	0.14	0.18	0.05	0.16
Collapses per placement (%)( $\downarrow$ )	0.49	0.48	0.12	0.42
Completion rate( $\uparrow$ )	0.00	0.67	0.95	0.84

TABLE III : Performance comparison on the base box set. All methods are evaluated using the 1M trained checkpoint over 50 episodes with randomized arrival order and identical seeds. Episode metrics average over evaluation episodes. Space utilization and termination rates are dimensionless; episode length counts placed boxes and collapses per placement is in percent. Metrics are defined in Sec. V-C . Completion rate is the fraction of episodes in which all boxes are placed.

	OL Mask ‡ [ 3 ]	GOPT [ 2 ]	GOPT+	SHORE(Ours)
Space utilization( $\uparrow$ )	0.242 $\pm$ 0.009	0.588 $\pm$ 0.017	0.715 $\pm$ 0.007	0.734 $\pm$ 0.006
Mean Episode Length( $\uparrow$ )	16.45 $\pm$ 1.07	41.08 $\pm$ 1.06	50.55 $\pm$ 0.69	50.83 $\pm$ 0.53
Unstable Rate( $\downarrow$ )	0.99 $\pm$ 0.01	0.26 $\pm$ 0.06	0.61 $\pm$ 0.08	0.53 $\pm$ 0.05
Collapses per placement (%)( $\downarrow$ )	5.69 $\pm$ 0.30	0.63 $\pm$ 0.15	1.20 $\pm$ 0.17	1.03 $\pm$ 0.10

TABLE IV : Performance on the expanded box set (12 geometries in Tab. II , 60 arrivals per episode). All methods are evaluated using the 1M trained checkpoint over 50 episodes with randomized arrival order and identical seeds. Learned methods report mean

\pm

standard deviation over three training seeds. No method completes the 60-box sequence, so the completion rate is 0 and omitted, and every episode that does not end in a collapse ends with no feasible placement (Sec. III-E ). Collapses per placement is in percent (Sec. V-C ).

Fig. 3 : Training curves on the base box set for the methods of Table III , up to 1M environment steps, computed from training episodes as in Fig. 4 . (a) Mean reward of training episodes (packed volume fraction). (b) Fraction of training episodes that end in a collapse. (c) Collapses per placement attempt (log scale). The original GOPT runs did not log how training episodes end, so GOPT appears only in (a).

Fig. 4 : Training curves on the expanded (FBA) box set. (a) Mean reward of training episodes (packed volume fraction), with actions sampled from each method’s training policy. (b) Fraction of training episodes that end in a collapse; the others end with no feasible placement. (c) Collapses per placement attempt during training (log scale), pooled over the training episodes in each window. All three panels are computed from training episodes. A method whose episodes collapse long before the bin fills stays near 1 in (b) even as its collapses per placement fall in (c). The curves end at 1M steps, the checkpoint evaluated in Table IV .

Panel of Figure 9 — Fig. 5 : An illustration of the scenes of one evaluation episode on the expanded box set, with the same box arrivals for all four methods, rendered in Isaac Gym after the episode ends. Each panel shows the final stack (left) and the table from above (right) with its stack and the boxes still waiting on the table. Pale boxes are light and dark boxes heavy; the red slab is the bin floor. OL Mask stopped when the last placement ended unstable and its collapse knocked boxes off the stack onto the table beside the bin. GOPT, GOPT stopped when on buffer boxes can find feasible placement. SHORE stopped when the last placement fails the stability test, but the unstability did not leads to stack collapse.

Fig. 6 : Real-robot packing with the UFactory Lite6. Key frames from the right camera view, in temporal order from left to right: the empty bin (yellow) and the stack after each of four successive placements. Boxes are picked from the table on the left of each frame.

Variant	Score	Collapse	AUROC	ECE
	( $\uparrow$ )	( $\downarrow$ )	( $\uparrow$ )	( $\downarrow$ )
SHORE	0.734	0.53	0.85	0.040
CW	0.708	0.60	0.86	0.051
AP	0.699	0.58	0.82	0.042
REPPO-D ∗	0.699	0.62	n/a	n/a

TABLE V : Ablation of where the stability estimate

S

acts, on the expanded box set. Each variant removes the named components from SHORE (Sec. V-E ); the variants that keep

S

use the default labels of Table VI . AUROC (higher:

S

ranks collapses below placements that hold more often) and expected calibration error (ECE; lower: the values of

S

are closer to the observed rate of holding) score the checkpoint’s

S

on held-out executed placements (Sec. V-C ); they are n/a for REPPO-D , which has no stability model to score.

Labels	$B_{s}$ / $B_{r}$	Score	Collapse	AUROC	ECE
		( $\uparrow$ )	( $\downarrow$ )	( $\uparrow$ )	( $\downarrow$ )
SHORE (default)	6/2	0.734	0.54	0.85	0.040
Executed only	0/0	0.731	0.52	0.77	0.009
1 simulated	1/0	0.716	0.52	0.78	0.004
8 random	0/8	0.732	0.52	0.79	0.035

TABLE VI : Ablation of the stability label source on the expanded box set, with both the actor prior and the critic weighting in place.

B_{s}

simulated labels per step are chosen by

\pi^{\gamma_{\pi}}\nu

and

B_{r}

uniformly at random; executed placements are always labeled.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Simulation
Parameter	Value
Buffer size $N$	$5$
Bin length and width $(L_{p},W_{p})$	$(25,25)$ cm
Maximum stacking height $H_{p}$	$20$ cm
Bin voxels $(l_{p},w_{p},h_{p})$	$(25,25,20)$ (1 cm bins)
Added placement noise	None

Appendix

TABLE VII : Experiment settings and hyperparameters grouped by function.

RL for Combinatorial Optimization

Reinforcement learning for combinatorial optimization learns sequential decisions and search strategies for difficult discrete problems.

See all 108 papers

Operationally Guided Placement-Aware Learning for Industrial Online 3D Bin Packing
Jul 30, 2026Dheeraj Poolavaram, Aanchal Rajesh Chugh, Sebastian DornRL for Combinatorial Optimization
TTZ Landsberg am Lech, Technische Hochschule Augsburg, Augsburg, Germany
HERB: Human-augmented Efficient Reinforcement learning for Bin-packing
Apr 23, 2025Gojko Perovic, Nuno Ferreira Duarte, Atabak Dehban +3Robotic RL Robotics
BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa, Italy, and with the Department of Excellence in Robotics and AI, Scuola Superiore Sant’Anna, Pisa, Italy. · Institute for Systems and Robotics, Instituto Superior Técnico, Universidade de Lisboa, 1049-001 Lisbon, Portugal.
Diffusion Reinforcement Learning Based Online 3D Bin Packing Spatial Strategy Optimization
Apr 13, 2026Jie Han, Tong Li, Qingyang Xu +3RL for Combinatorial Optimization
School of Mechanical, Electrical, and Information Engineering, Shandong University, Weihai 264209, China · Shandong Key Laboratory of Intelligent Electronic Packaging Testing and Application, Shandong University, Weihai 264209, China

Jul 30, 2026J/K move · Enter open · S save

Operationally Guided Placement-Aware Learning for Industrial Online 3D Bin Packing

Dheeraj Poolavaram, Aanchal Rajesh Chugh, Sebastian Dorn

TTZ Landsberg am Lech, Technische Hochschule Augsburg, Augsburg, Germany

RL for Combinatorial Optimization

Open paper AskPDF arXiv

The online three-dimensional bin packing problem (3D-BPP) is a longstanding challenge in logistics and industrial palletizing. Recent learning-based methods use a learned policy to select among feasible candidate placements. Performance depends on the candidate generator and representation, especially in industrial settings where packings must be space-efficient, stable, compact, and balanced. However, prior work has mainly optimized the policy, while candidate generation and representation remain largely geometry-driven. We address this gap with OPAL, an operationally guided placement-aware learning framework for industrial online 3D-BPP which combines an Operationally Guided Empty-Maximal-Space generator (OG-EMS), an operational representation for each candidate placement, and a masked ranking policy trained with proximal policy optimization. OG-EMS evaluates multiple anchors within each free-space region and prioritizes low, well-supported, compact, and spatially diverse placements. An xLSTM-based Placement Encoder models dependencies among geometric and operational candidate attributes, while a lightweight recurrent core combines the resulting embeddings with the current item and pallet state to rank feasible actions. On the BED-BPP benchmark, OPAL achieves a mean space utilization of 0.49, with improvements of 15.1% from operationally guided candidate generation and 6.3% from learned ranking, while maintaining robust inference-time performance.

Abstract

RL for Combinatorial Optimization

Explore similar work

Operationally Guided Placement-Aware Learning for Industrial Online 3D Bin Packing

HERB: Human-augmented Efficient Reinforcement learning for Bin-packing

Diffusion Reinforcement Learning Based Online 3D Bin Packing Spatial Strategy Optimization