Preference-Conditioned Reinforcement Learning for Space-Time Efficient Online 3D Bin Packing

Nikita Sarawgi; Omey M. Manyar; Fan Wang; Thinh H. Nguyen; Daniel Seita; Satyandra K. Gupta

cs.ROMar 8, 2026

Preference-Conditioned Reinforcement Learning for Space-Time Efficient Online 3D Bin Packing

Authors: Nikita Sarawgi, Omey M. Manyar, Fan Wang, Thinh H. Nguyen, Daniel Seita, Satyandra K. Gupta

Organizations: Viterbi School of Engineering, University of Southern California, Los Angeles, CA, USA. · Amazon Robotics, North Reading, MA, USA.

Abstract

Robotic bin packing is widely deployed in warehouse automation, with current systems achieving robust performance through heuristic and learning-based strategies. These systems must balance compact placement with rapid execution, where selecting alternative items or reorienting them can improve space utilization but introduce additional time. We propose a selection-based formulation that explicitly reasons over this trade-off: at each step, the robot evaluates multiple candidate actions, weighing expected packing benefit against estimated operational time. This enables time-aware strategies that selectively accept increased operational time when it yields meaningful spatial improvements. Our method, STEP (Space-Time Efficient Packing), uses a preference-conditioned, Transformer-based reinforcement learning policy, and allows generalization across candidate set sizes and integration with standard placement modules. It achieves a 44% reduction in operational time without compromising packing density. Additional material is available at https://step-packing.github.io.

Figures & tables

Figure 1: Visualization of the semi-online 3D bin packing problem (3D-BPP) with a buffer. The robot grasps the box from the front-face over the top-face. It reorients it to a transportable configuration and places it in the bin. The selected face impacts the placement position in the bin.

Figure 2: Overall architecture of STEP , a preference-conditioned Transformer-based policy network. The inputs consist of the bin state, item–face buffer with geometric and temporal attributes, and the current preference vector. These are embedded and processed by stacked attention-based encoders called Transformer-Select. The actor outputs logits over item–face candidates conditioned on the preference vector, while the preference-conditioned critic predicts vector-valued returns for space utilization and operational time, aligned with the multi-objective reward structure.

Figure 3: The Transformer-Select module used in our proposed method STEP .

Figure 4: Pareto front of evaluated policies in the space–time trade-off space (left). Each point represents a distinct policy configuration, with the frontier highlighting non-dominated solutions balancing space utilization and operational time. Packing visualizations for STEP-1 (i) and STEP-3 (ii) at three preference vectors A = $[0.02,0.98]$ , B = $[0.53,0.47]$ , and C = $[0.95,0.05]$ are shown (right), where the first element weights space and the second weights time. The corresponding $\omega$ values are marked with red circles in the Pareto front.

Method	Uti	Var	Num	Time	VarTime	Top $\%$	Front $\%$	Left+Right $\%$	Back $\%$
TopFaceSpace	75.59	7.25	28.95	57.87	14.31	100.00	0.00	0.00	0.00
ReorientSpace-1	84.05	5.64	32.26	96.60	20.01	32.87	34.45	32.67	0
ReorientTime-1	75.98	6.97	29.07	36.08	9.30	59.07	27.04	12.16	1.73
STEP-1 (Ours)	81.76	6.16	31.55	53.84	12.46	49.79	26.50	20.77	2.92

Table I : Comparison of bin packing performance across space and time metrics for different methods.

Method	Uti	Var	Num	Time	VarTime
STEP-1	81.76	6.16	31.55	53.84	12.46
STEP-3	87.45	3.82	34.10	54.89	12.01
STEP-5	89.72	3.21	34.96	53.89	11.68

Table II: Generalization performance of our proposed method across different buffer sizes, evaluated on space utilization and operational time using a fixed preference vector

\omega=(0.95,0.05)

. The suffix indicates the buffer size.

Figure 5: Average Space Utilization ( $\%$ ) and Average Time Taken under increasing variability in item box sizes.

Method	Uti	Var	Num	Time	VarTime
GOPT_var1	83.01	6.72	31.78	95.59	19.80
ReorientSpace-1	84.05	5.64	32.26	96.60	20.01
MCTS-5	54.58	18.55	13.41	46.95	19.21
ReorientSpace-5	91.26	3.35	35.71	106.71	20.12
STEP-5	89.72	3.21	34.96	53.89	11.68

Table III: Comparison of space and time performance across MCTS, GOPT variants, ReorientSpace, and STEP-5.

Panel of Figure 6 — Figure 6: The real-world experiments with an ABB robot and a suction-cup end effector. The system maintains a buffer of size 3. (a) The robot grasps the box by its front face, (b) reorients it to a transportable state and (c) places it in the bin at the predicted location.

Robotics

Robotics studies the design and control of machines that perceive, move, and act in physical environments.

See all 751 papers

HERB: Human-augmented Efficient Reinforcement learning for Bin-packing
Apr 23, 2025Gojko Perovic, Nuno Ferreira Duarte, Atabak Dehban +3Robotic RL Robotics
BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa, Italy, and with the Department of Excellence in Robotics and AI, Scuola Superiore Sant’Anna, Pisa, Italy. · Institute for Systems and Robotics, Instituto Superior Técnico, Universidade de Lisboa, 1049-001 Lisbon, Portugal.
Operationally Guided Placement-Aware Learning for Industrial Online 3D Bin Packing
Jul 30, 2026Dheeraj Poolavaram, Aanchal Rajesh Chugh, Sebastian DornRL for Combinatorial Optimization
TTZ Landsberg am Lech, Technische Hochschule Augsburg, Augsburg, Germany
Spatial Heuristic Strategy based Diffusion Reinforcement Learning for Online 3D Bin Packing
Oct 23, 2026Tong Li, Qingyang XuReinforcement Learning RL for Combinatorial Optimization
School of Airspace Science and Engineering, Shandong University, Weihai 264209, China, and also with the Shandong Key Laboratory of Intelligent Electronic Packaging Testing and Application, Shandong University, Weihai 264209.

Apr 23, 2025J/K move · Enter open · S save

HERB: Human-augmented Efficient Reinforcement learning for Bin-packing

Gojko Perovic, Nuno Ferreira Duarte, Atabak Dehban, Gonçalo Teixeira+2

BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa, Italy, and with the Department of Excellence in Robotics and AI, Scuola Superiore Sant’Anna, Pisa, Italy. · Institute for Systems and Robotics, Instituto Superior Técnico, Universidade de Lisboa, 1049-001 Lisbon, Portugal.

Robotic RL Robotics Robotic Manipulation Learning from Demonstration

Open paper AskPDF arXiv

Packing objects efficiently is a fundamental problem in logistics, warehouse automation, and robotics. When dealing with highly diverse 3D objects (household or grocery items), closed-form solutions are infeasible, and heuristic or model-free Reinforcement Learning~(RL) methods tend to focus solely on geometric optimization, relying on exhaustive searches of the discretized solution space. This leads to long training times (for pure RL) and high latency (heuristics), limited transferability to robotic scenarios, and ultimately ignores object characteristics (fragility, deformability) and human preferences. We propose HERB, a human-augmented RL framework for packing irregular objects, the first to explore the potential of learning from human demonstrations to solve this complex task. It leverages human demonstrations of packing strategies, which inherently exhibit latent factors such as space optimization, stability, and object properties that are difficult to model explicitly. The human-expert data is combined with RL exploration to provide the placement of each object inside the container. Experimental results show that our method outperforms heuristic, purely RL-based, and imitation learning approaches in packing efficiency and latency. Qualitative results highlight that our packing strategy produces more stable, human-like arrangements, which we expect to be more appropriate and widely accepted. Finally, we demonstrate the real-world feasibility of our method on a robotic system.

Abstract

Robotics

Explore similar work

HERB: Human-augmented Efficient Reinforcement learning for Bin-packing

Operationally Guided Placement-Aware Learning for Industrial Online 3D Bin Packing

Spatial Heuristic Strategy based Diffusion Reinforcement Learning for Online 3D Bin Packing