Gibson Env: Real-World Perception for Embodied Agents

Fei Xia; Amir Zamir; Zhi-Yang He; Alexander Sax; Jitendra Malik; Silvio Savarese

cs.AIAug 31, 2018

Gibson Env: Real-World Perception for Embodied Agents

Authors: Fei Xia, Amir Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, Silvio Savarese

Organizations: Stanford University · University of California, Berkeley

Abstract

Developing visual perception models for active agents and sensorimotor control are cumbersome to be done in the physical world, as existing algorithms are too slow to efficiently learn in real-time and robots are fragile and costly. This has given rise to learning-in-simulation which consequently casts a question on whether the results transfer to real-world. In this paper, we are concerned with the problem of developing real-world perception for active agents, propose Gibson Virtual Environment for this purpose, and showcase sample perceptual tasks learned therein. Gibson is based on virtualizing real spaces, rather than using artificially designed ones, and currently includes over 1400 floor spaces from 572 full buildings. The main characteristics of Gibson are: I. being from the real-world and reflecting its semantic complexity, II. having an internal synthesis mechanism, "Goggles", enabling deploying the trained models in real-world without needing further domain adaptation, III. embodiment of agents and making them subject to constraints of physics and space.

Figures & tables

Figure 2: Overview of our view synthesis pipeline. The input is a sparse set of RGB-D Panoramas with their global camera pose. (a,b) Each RGB-D panorama is projected to the target camera pose and rendered. (b) View Selection determines from which panorama each target pixel should be picked, favoring panoramas that provide denser pixels for each region. (c) The pixels are selected and local gaps are interpolated with bilinear sampling. (d) A neural network ( $f$ ) takes in the interpolated image and fills in the dis-occluded regions and fixes artifacts.

Figure 3: Loss configuration for neural network based view synthesis. The loss contains two terms. The first is to transform the renderings to ground truth target images. The second is to alter ground truth target images to match the transformed rendering. A sample case is shown. (Best viewed on screen and zoomed-in.)

Figure 4: Physics Integration and Embodiment. A Mujoco humanoid model is dropped onto a stairway demonstrating a physically plausible fall along with the corresponding visual observations by the humanoid’s eye. The first and second rows show the physics engine view of 4 sampled time steps and their corresponding rendered RGB views, respectively.

Figure 5: Sample spaces in Gibson database. The spaces are diverse in terms of size, visuals, and function, e.g. businesses, construction sites, houses. Upper: Sample 3D models. Lower: Sample images from Gibson database (left) and some of other environments [ 27 , 46 , 71 , 82 , 48 , 99 , 33 , 105 ] (right).

Figure 6: Qualitative results of view synthesis and Goggles . Top to bottom rows show images before neural network correction, after neural network correction, target image seen through Goggles , and target image (i.e. ground truth real image). The first column shows a pano and the rest are sample zoomed-in patches. Note the high similarity between $2^{nd}$ and $3^{rd}$ row, signifying the effectiveness of Goggles . (Best viewed on screen and zoomed-in.)

Dataset	Gibson	SUNCG	Matterport3D
Number of Spaces	572	45622	90
Total Coverage $m^{2}$	211k	5.8M	46.6K
SSA	1.38	0.74	0.92
Nav. Complexity	5.98	2.29	7.80
Real-World Transfer Err	0.92 §	2.89 †	2.11 †

Table 1: Benchmarking Space Databases: Comparison of Gibson database with SUNCG [ 87 ] (hand designed synthetic), and Matterport3D [ 14 ] . § Rendered with Gibson, † rendered with MINOS [ 75 ] .

Resolution	128x128	256x256	512x512
RGBD, pre network $f$	109.1	58.5	26.5
RGBD, post network $f$	77.7	30.6	14.5
RGBD, post small network $f$	87.4	40.5	21.2
Depth only	253.0	197.9	124.7
Surface Normal only	207.7	129.7	57.2
Semantic only	190.0	144.2	55.6

Table 2: Rendering speed (FPS) of Gibson on a single GPU for different resolutions and output configurations. Tested on E5-2697 v4 with Tesla V100 in headless rendering mode. As a faster setup (“small network”), we also trained a smaller filler network with donwsized input geometric renderings. This setup achieves a higher FPS at the expense of inferior visual quality compared to full-size filler network.

Train	Test	Static Tasks		Appearance
Train	Test	Scene Class Acc.	Depth Est. Error	SSIM	L1
$\mathcal{I}_{s}$	$\mathcal{I}_{t}$	0.280	1.026	0.627	0.096
$f(\mathcal{I}_{s})$	$\mathcal{I}_{t}$	0.266	1.560	0.480	0.10
$f(\mathcal{I}_{s})$	$u(\mathcal{I}_{t})$	0.291	0.915	0.816	0.051

Table 3: Evaluation of view synthesis and transferring to real-world. Static Tasks column shows on both scene classification task and depth estimation tasks, it is easiest to transfer from

f(\mathcal{I}_{s})

u(\mathcal{I}_{t})

compared with other cross-domain transfers. Appearance columns compare L1 and SSIM distance metrics for different pairs showing the combination of network

f

and Goggles

u

achieves best results.

Figure 7: Evaluation of transferring to real-world from Gibson. (a) Error of depth estimation for all train-test combinations. (b,c) MMD and CORAL distributional distances. All tests are in support of Goggles .

Panel of Figure 8 — Figure 8: Visual Local planning and obstacle avoidance. Reward curves for perceptual vs non-perceptual husky agents and a sample trajectory.

Figure 9: Distant Visual Navigation. The initial locations and target are shown. The agent succeeds only when provided with visual inputs.

Figure 1: Additional view synthesis results. We provide additional view synthesis and domain adaptation results. The format is similar to Fig. 6 in the main paper. The results have high resolution and are very rich in details, so we encourage readers to zoom in to see details.

Type	Kernel	Dilation	Stride	Output
conv.	$5\times 5$	1	$1\times 1$	6
conv.	$5\times 5$	1	$2\times 2$	$n_{f}$
conv.	$3\times 3$	1	$1\times 1$	$n_{f}$
conv.	$5\times 5$	1	$2\times 2$	$4n_{f}$
dilated conv.	$3\times 3$	1	$1\times 1$	$4n_{f}$
dilated conv.	$3\times 3$	1	$1\times 1$	$4n_{f}$

Table 1: Network Architecture. The network architecture is detailed in this table. We use three types of layers: convolutional layers(conv.), dilated convolutional layers(dilated conv.) and deconvolutional layers(deconv.). The network size is configurable with parameter

n_{f}

Figure 2: Sample frames with semantic annotation . Top: NN filled images; Bottom: Corresponding semantic labels of pixels.

Figure 3: Ant climbing downstairs tasks reward curve . Left: ending position of sensor-only (red) and perceptual (green) agent. Right: reward curves and test score. Here we plot the reward curve over the first 700K frames when the agent starts to acquire stair climbing ability.

Embodied AI

Embodied AI studies agents that perceive, reason, and act through bodies in physical or simulated environments.

See all 255 papers

ActiveGlasses: Learning Manipulation with Active Vision from Ego-centric Human Demonstration
Apr 9, 2026Yanwen Zou, Chenyang Shi, Wenye Yu +5Active Vision for Robot Manipulation Robot Imitation Learning
Shanghai Jiao Tong University · Shanghai Innovation Institute · Noematrix Ltd.
Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation
Jul 15, 2026Boyu Mi, Mengchen Ma, Yifei Yao +10Mobile Manipulation Sim-to-Real Transfer
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Southeast University +4
ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception
Oct 3, 2026Ruoxuan Feng, Yutong Chen, Ruihua Song +4Robotic Perception Multimodal Sensor Fusion
Gaoling School of Artificial Intelligence, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Beijing Academy of Artificial Intelligence +3

Apr 9, 2026J/K move · Enter open · S save

ActiveGlasses: Learning Manipulation with Active Vision from Ego-centric Human Demonstration

Yanwen Zou, Chenyang Shi, Wenye Yu, Han Xue+4

Shanghai Jiao Tong University · Shanghai Innovation Institute · Noematrix Ltd.

Active Vision for Robot Manipulation Robot Imitation Learning Cross-Embodiment Robot Learning Zero-Shot Robotic Manipulation Robotic Data Collection

Open paper AskPDF arXiv

Large-scale real-world robot data collection is a prerequisite for bringing robots into everyday deployment. However, existing pipelines often rely on specialized handheld devices to bridge the embodiment gap, which not only increases operator burden and limits scalability, but also makes it difficult to capture the naturally coordinated perception-manipulation behaviors of human daily interaction. This challenge calls for a more natural system that can faithfully capture human manipulation and perception behaviors while enabling zero-shot transfer to robotic platforms. We introduce ActiveGlasses, a system for learning robot manipulation from ego-centric human demonstrations with active vision. A stereo camera mounted on smart glasses serves as the sole perception device for both data collection and policy inference: the operator wears it during bare-hand demonstrations, and the same camera is mounted on a 6-DoF perception arm during deployment to reproduce human active vision. To enable zero-transfer, we extract object trajectories from demonstrations and use an object-centric point-cloud policy to jointly predict manipulation and head movement. Across several challenging tasks involving occlusion and precise interaction, ActiveGlasses achieves zero-shot transfer with active vision, consistently outperforms strong baselines under the same hardware setup, and generalizes across two robot platforms.

Abstract

Embodied AI

Explore similar work

ActiveGlasses: Learning Manipulation with Active Vision from Ego-centric Human Demonstration

Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation

ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception