OMAR: Open-World Multi-Attribute Reasoning for Scene-Conditioned Shelf Organization

Benedikt Kreis; Rohit Menon; Benno Wingender; Muhammad Ehsan Ullah; Nils Dengler; Maren Bennewitz

cs.ROOct 21, 2026

OMAR: Open-World Multi-Attribute Reasoning for Scene-Conditioned Shelf Organization

Authors: Benedikt Kreis, Rohit Menon, Benno Wingender, Muhammad Ehsan Ullah, Nils Dengler, Maren Bennewitz

Organizations: Humanoid Robots Lab, University of Bonn, Germany. · Center for Robotics, University of Bonn, Germany. · Lamarr Institute for ML and AI, Bonn, Germany. · Interactive Robot Perception and Learning Lab, Technical University of Darmstadt, Germany.

Abstract

With the advent of foundation models, robots are increasingly capable of understanding and executing explicitly specified human tasks. Extending these capabilities to open-ended tasks, however, requires robots to infer appropriate actions from the observed scene. Household organization exemplifies this challenge, as robots must often act without detailed instructions, predefined object vocabularies, or known object-receptacle associations. While robots may follow instructions or learn organization preferences from demonstrations, inferring the latent organization of an occasionally changing scene and placing a novel object consistently with it remains challenging. We propose an open-world multi-attribute reasoning pipeline that grounds unknown objects across views and constructs an instance-level 3D scene graph for placement reasoning. First, a vision-language model proposes labels together with attributes such as content and packaging type, which are grounded through segmentation, depth, and multi-view association. Then, an LLM reasons over this representation to infer the latent shelf organization and rank scene-consistent object placements without explicitly user-provided preferences. The resulting placement rankings guide subsequent geometric and kinematic feasibility checks for execution. Our experimental evaluation shows that our approach leads to improved instance mapping and placement coherence over single-view language-grounding and geometry-only baselines, enabling autonomous placement of previously unseen objects without predefined object-receptacle mappings or class-specific retraining.

Figures & tables

Fig. 1 : The robot’s task is to pick an unknown object from the table and find the semantically most coherent and geometrically feasible placement in the shelving unit across multiple shelf boards (top left). Our system creates a 3D representation from multiple views (bottom left) to predict placement poses considering multiple object attributes (right). Our approach combines open-world multi-attribute and multi-view instance mapping to generate a 3D semantic scene graph, which enables scene-conditioned placement reasoning with feasibility-aware grasp-placement planning to execute placements of previously unseen objects into space-limited shelving units.

Fig. 2 : Overview of the proposed approach: First the robot discovers the environment (A) by moving the camera to identify the unknown grasp object presented on the table and then to the shelf to create an object instance map (B) using descriptions and masks of each object from the view set. Based on the instance map, we create a 3D scene graph, containing semantic and geometric attributes of each object. Furthermore, we create a 3D model of the target object, to estimate its pose and evaluate valid grasps (C). Afterwards, we predict placement candidates across all shelf boards (D) by sampling geometrically possible candidates on the map and match them with semantically suitable candidates predicted based on the 3D semantic scene graph. The matched candidates are ranked and based on multiple feasibility checks unsuitable candidates are discarded. Finally, a motion planner calculates collision-free trajectories for the remaining candidates (grasp-placement pairs) and executes the highest-scored one.

Fig. 3 : Based on the 3D semantic instance map of the shelving unit, we generate a 3D semantic scene graph representation (top right), which contains detailed semantic and geometrical attributes of each object instance. We assign the same attributes to the target object based on the RGB-D observation from the table (top left). Providing these inputs to the LLM, together with the prediction query (bottom left), it predicts ranked placement candidates (bottom right).

Metric

Single Row Scenes

Multi Row Scenes

Varying Clearance Scenes

GT-Labels

OURS

Basic

BL1

GT-Labels

OURS

Basic

BL1

GT-Labels

OURS

Basic

BL1

a) F1-Label [%] ↑

79\pm 1.4

\mathbf{78}\pm 1.3

71\pm 1.7^{\star}

36\pm 2.1^{\star\star\star}

64\pm 1.3

\mathbf{63}\pm 1.3

58\pm 1.5^{\star}

24\pm 1.6^{\star\star\star}

58\pm 1.5

\mathbf{56}\pm 1.6

51\pm 1.4^{\star}

24\pm 1.5^{\star\star\star}

b) BB-IoU [%] ↑

63\pm 0.8

\mathbf{63}\pm 0.9

62\pm 1.2

21\pm 0.1^{\star\star\star}

41\pm 0.7

\mathbf{41}\pm 0.7

39\pm 1.0

20\pm 0.2^{\star\star\star}

40\pm 1.1

\mathbf{40}\pm 1.0

39\pm 1.2

21\pm 0.1^{\star\star\star}

c) Instances [%] ↑

97\pm 0.8

\mathbf{96}\pm 0.7

\mathbf{96}\pm 1.4

49\pm 1.9^{\star\star\star}

80\pm 1.0

\mathbf{80}\pm 0.8

\mathbf{80}\pm 1.6

29\pm 1.2^{\star\star\star}

85\pm 1.0

\mathbf{83}\pm 1.0

\mathbf{83}\pm 1.3

31\pm 1.4^{\star\star\star}

TABLE I : Instance mapping is evaluated over 100 runs per scene dataset using canonical-label F1, 3D bounding-box overlap (BB-IoU), and the number of mapped instances relative to the ground truth (Instances). GT-Labels provides an upper-bound reference by evaluating instance mapping with ground-truth object labels instead of predicted labels. OURS outperforms BL1 [ 1 ] in all three metrics across all scene sets despite using the same SAM 3 segmenter, supporting the combination of richer semantic descriptions and multi-view evidence. OURS also maps substantially more instances than BL1 across all scene sets, reaching

80\text{\,}\mathrm{\%}

of the ground-truth instance count in multi-row scenes compared with

29\text{\,}\mathrm{\%}

for BL1. BL1’s noun-based label extraction likely contributes to the gap by discarding descriptive cues needed for object disambiguation. Compared with the attribute-free Basic variant, OURS improves F1 and matches or improves BB-IoU, supporting attribute-based instance association.

\star

and

\star\star\star

denote statistically significant results for our approach compared to all other approaches with

p

-value thresholds of

0.05

and

0.001

respectively.

Metric	Single Row Scenes			Multi Row Scenes			Varying Clearance Scenes
Metric	OURS	BL1	BL2	OURS	BL1	BL2	OURS	BL1	BL2
a) SR [%] ↑	$96\pm 2.0$	$20\pm 4.0^{\star\star\star}$	$\mathbf{97}\pm 1.7$	$\mathbf{95}\pm 2.2$	$20\pm 2.7^{\star\star\star}$	$\mathbf{95}\pm 2.1$	$\mathbf{81}\pm 3.9$	$5\pm 2.2^{\star\star\star}$	$\mathbf{81}\pm 2.9$
b) SF-EQ [%] ↑	$\mathbf{60}\pm 1.5$	$35\pm 2.6^{\star\star\star}$	$52\pm 1.6^{\star\star\star}$	$\mathbf{63}\pm 1.5$	$34\pm 2.7^{\star\star\star}$	$53\pm 1.6^{\star\star\star}$	$\mathbf{61}\pm 2.2$	$37\pm 3.0^{\star\star\star}$	$50\pm 2.1^{\star\star\star}$
c) SF-LLM [%] ↑	$\mathbf{58}\pm 1.7$	$32\pm 2.6^{\star\star\star}$	$47\pm 1.9^{\star\star\star}$	$\mathbf{58}\pm 1.6$	$29\pm 2.5^{\star\star\star}$	$46\pm 1.8^{\star\star\star}$	$\mathbf{58}\pm 2.2$	$33\pm 2.8^{\star\star\star}$	$45\pm 2.2^{\star\star\star}$

TABLE II : Results measure execution success (SR) and semantic fit with equal attribute weights (SF-EQ) and scene-dependent LLM weights (SF-LLM) based on 100 runs per scene set. OURS achieves the highest SF-EQ and SF-LLM across all scenes while retaining a success comparable to the geometry-only baseline BL2. BL1 achieves substantially lower success, SF-EQ, and SF-LLM, consistent with its limited instance coverage providing fewer object anchors for placement reasoning.

\star\star\star

denotes statistically significant results for our approach compared to BL1 and BL2 with

p

-value threshold of

0.001

Attribute	Content		Packaging		Edible		Orientation
Attribute	OURS	Score	OURS	Score	OURS	Score	OURS	Score
SF-LLM [%] ↑	$\mathbf{72}\pm 2.2$	$68\pm 2.4^{\star}$	$\mathbf{83}\pm 2.1$	$78\pm 2.1^{\star}$	$\mathbf{70}\pm 2.0$	$66\pm 2.2^{\star}$	$\mathbf{51}\pm 3.8$	$41\pm 3.6^{\star\star}$

TABLE III : We determine attribute weights with three independent LLM experts using four sets of ten scenes, each dominated by one attribute while randomizing the others. Using these weights on 20 unseen scene sets, we compare our LLM-based predictor (OURS), against an equally weighted score-based predictor (Score). The results show that our predictor captures scene-specific attribute importance and produces more attribute-coherent placements without manual weight tuning.

\star

and

\star\star

denote statistically significant results for our approach compared to the score-based predictor with

p

-value thresholds of

0.05

and

0.01

respectively.

Panel of Figure 8 — Fig. 5 : We show an exemplary real-world rollout of the robot picking and placing a previously unseen object into a shelf. Target and shelf observations show segmentation results (top) and the corresponding robot configurations (bottom). The target object mesh supports geometric feasibility assessment, while the 3D semantic scene graph resulting from the depicted instance map encodes observed object attributes and spatial relations. Given this graph and the target attributes, the LLM ranks placements according to the inferred shelf organization. The executed top-ranked placement groups the target object with shelf objects of similar content and the same packaging type, consistent with the local organization of the shelf.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,671 papers

Aug 21, 2026

presto: Efficient, Training-free, and Open-world Object Placement via Imaginary Search

Object placement is critical in image composition, requiring spatially and semantically coherent positioning of objects within diverse scenes. Existing approaches typically rely on hand-crafted rules or supervised learning on limited datasets, which restricts their generalization and interpretability, especially in open-world scenarios involving novel objects and scenes. In this work, we reformulate open-world object placement as a heuristic search task guided by reasoning from a Multimodal Large Language Model (MLLM). We introduce presto, a zero-shot, training-free framework that operates within an imaginary action space to iteratively refine object position and scale. Our coarse-to-fine search strategy ensures fast convergence, and we evaluate two decision-making variants: Metric-guided Selection and MLLM-as-a-judge. Experiments across multiple benchmarks show that presto~achieves state-of-the-art performance, particularly in previously unseen, open-world settings. Human studies further reveal that the MLLM-as-a-judge variant produces more perceptually coherent placements than metric-driven approaches, highlighting a gap between standard evaluation metrics and human visual judgment.

Weixuan Ding, Shang Liu, Hanyu Pei +1

Wuhan University, Wuhan Hubei Province, China · University of Louisville, Louisville KY, USA

MLLM Reasoning Compositional Image Generation

Ask PDF

Jun 15, 2026

Abstention-Aware Personalized Object Rearrangement via Uncertainty-Guided LLM Assistance

Robotic assistance in household environments requires not only predicting where objects should be placed, but also reasoning about when objects should not be placed at all. Existing approaches to personalized object rearrangement primarily focus on placement decisions under the assumption of clean observations and complete actionability, limiting their applicability in realistic, cluttered, and partially erroneous settings. In this paper, we introduce APOLLO, a hybrid framework for abstention-aware personalized object rearrangement that combines a lightweight, personalized embedding model (PEM) with selective large language model (LLM) assistance. PEM is trained for each user-environment pair using a small number of demonstrations, operates entirely on CPU, and produces uncertainty estimates, which are used to selectively invoke LLM-based reasoning only for ambiguous decisions, balancing efficiency, privacy, and reasoning capability. To evaluate this formulation beyond existing benchmarks, we introduce APOR, a synthetic, LLM-generated dataset that captures room-level, multi-furniture environments, diverse organizational profiles, explicit abstention behavior, and noisy partial scene context. Extensive experiments on both PARSEC and APOR provide initial evidence that APOLLO improves over prior LLM-based baselines in controlled benchmark settings while substantially reducing LLM usage. Code is available at https://github.com/PaInt-Lab/APOLLO.

Sam Collin, Ali Ayub

Concordia University, Montreal, Quebec H3G 1M8, Canada

Selective Prediction Robot Task Planning Robot Manipulation Benchmarks

Ask PDF

Oct 20, 2026

GALA: Geometry-Aware Language Model for Controllable Object Arrangement

Reasoning about fine-grained placement constraints, such as support and containment, is essential for realistic 3D scene creation. It remains a significant challenge for existing methods that rely on simplified object representations like oriented bounding box (OBB), which are blind to the detailed geometry. To address this limitation, we introduce GALA, a geometry-aware language model with explicit point cloud perception for controllable object placement. Our model performs autoregressively by predicting the 6D pose of one target object at a time conditioned on the scene context and placement instruction. We utilize a two-stage training strategy: the first stage aligns geometric and textual modalities through pretraining tasks, and the second stage fine-tunes the model for placement. To facilitate this task, we also curate a multi-source scene dataset covering diverse room types and fine-grained furniture arrangements, which will be open-sourced for community benefit. Experiments on our benchmark show our model significantly outperforms baselines without explicit geometric reasoning. We further develop a multi-stage plan-place-verify agent pipeline that uses the model for text-driven scene generation. Project Page: https://xccelephant.github.io/paper/gala/

Chucheng Xiang, Runze Wang, Zhi Deng +10

University of Science and Technology of China, Hefei, China · Tencent, Shenzhen, China · Tsinghua University, Shenzhen, China +1

3D Scene Layout Generation 3D VLMs Indoor Scene Generation Autoregressive 3D Scene Generation

Ask PDF

Abstract

Robotic Manipulation

Explore similar work

presto: Efficient, Training-free, and Open-world Object Placement via Imaginary Search

Abstention-Aware Personalized Object Rearrangement via Uncertainty-Guided LLM Assistance

GALA: Geometry-Aware Language Model for Controllable Object Arrangement