Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering

Sebin Lee; Jumin Lee; Taeyeon Kim; Younju Na; Woobin Im; Sung-Eui Yoon

cs.ROMar 27, 2026

Visual-RRT: Finding Paths toward Visual-Goals via Differentiable Rendering

Authors: Sebin Lee, Jumin Lee, Taeyeon Kim, Younju Na, Woobin Im, Sung-Eui Yoon

Organizations: 1KAIST · 2Samsung Electronics

Abstract

Rapidly-exploring random trees (RRTs) have been widely adopted for robot motion planning due to their robustness and theoretical guarantees. However, existing RRT-based planners require explicit goal configurations specified as numerical joint angles, while many practical applications provide goal specifications through visual observations such as images or demonstration videos where precise goal configurations are unavailable. In this paper, we propose visual-RRT (vRRT), a motion planner that enables visual-goal planning by unifying gradient-based exploitation from differentiable robot rendering with sampling-based exploration from RRTs. We further introduce (i) a frontier-based exploration-exploitation strategy that adaptively prioritizes visually promising search regions, and (ii) inertial gradient tree expansion that inherits optimization states across tree branches for momentum-consistent gradient exploitation. Extensive experiments across various robot manipulators including Franka, UR5e, and Fetch demonstrate that vRRT achieves effective visual-goal planning in both simulated and real-world settings, bridging the gap between sampling-based planning and vision-centric robot applications. Our code is available at https://sgvr.kaist.ac.kr/Visual-RRT.

Figures & tables

Figure 1: Motion planning toward visual goals. (a) RRT planners efficiently explore the C-space from the start configuration $q_{\text{start}}$ yet require explicit goal configurations $q_{\text{goal}}$ , limiting their direct use with image-specified goals. (b) Visual gradient-based methods minimize the rendering loss with respect to the goal image $I_{\text{goal}}$ , but often struggle to reach the desired configuration. (c) Our vRRT integrates sampling-based exploration and visual-gradient exploitation, enabling the planner to efficiently discover a path toward $I_{\text{goal}}$ . Gray regions in (a–c) indicate unreachable parts of the C-space. (d) Planning progression over time: while gradient-based optimization stagnates in local minima (top), our method continues to explore and successfully reaches the visual goal (bottom). In each step of our visualization, multiple robot poses represent parallel exploration and exploitation directions sampled during a single expansion step.

Figure 2: Overview of visual-RRT (vRRT). vRRT incrementally grows a search tree from step $t$ to $t+1$ toward a visual goal $I_{\text{goal}}$ by integrating sampling-based exploration with gradient-based exploitation. Frontier-based steering: At each iteration, visually promising nodes (cyan) are prioritized as parents via a truncated geometric distribution $p_{\text{frontier}}(k)$ to balance goal-directed exploitation with exploration. (Exploration) Random steering expands the tree toward randomly sampled configurations from $\text{Ball}(0,\rho)$ around selected frontier nodes for broad C-space coverage. (Exploitation) Visual-gradient steering leverages differentiable rendering: parent configurations $q$ are rendered to compute visual loss $\mathcal{L}_{\text{render}}$ against $I_{\text{goal}}$ , yielding visual gradient $\nabla_{q}\mathcal{L}_{\text{render}}$ for goal-directed updates. We introduce inertial gradient expansion where each node stores optimization states (first moment $m_{p}$ , second moment $v_{p}$ , iteration $i_{p}$ ) inherited from its parent, enabling independent momentum-based gradient trajectories across multiple branches toward the visual goal.

Robot	Method	Metric	Target distance bins (rad)					Avg.
Robot	Method	Metric	$0.5$	$1.0$	$1.5$	$2.0$	$2.5$	Avg.
Franka	Dr.Robot	SR	59.8	21.5	9.5	4.2	1.5	19.3
		Time	9.08	9.67	9.57	12.41	10.52	10.25
		PL	2.75	3.59	4.26	5.32	6.06	4.40
	Prof.Robot	SR	71.2	27.3	12.2	5.3	2.0	23.6
		Time	12.24	13.14	13.22	15.48	13.98	13.61

Table 1: Visual-goal motion planning results. Success rate (SR, %), planning time (Time, s), and path length (PL, rad) across configuration-space distance bins. vRRT achieves substantially higher success rates, particularly at larger distances. Time and PL are averaged over successful trials; vRRT solves a broader range of problem difficulties compared to baselines.

Figure 3: Qualitative visual-goal motion planning results. Top three rows show trajectories (yellow) from Dr.Robot [ 30 ] , Prof.Robot [ 42 ] , and vRRT; bottom row shows start/goal poses and RRT ∗ reference (red). While all methods succeed at small distances, Dr.Robot produces circuitous trajectories and frequently fails at larger distances due to local minima under occlusions. Prof.Robot reduces path curvature but remains less effective than vRRT, which consistently discovers collision-free paths with geometric structure similar to RRT ∗ solutions.

Figure 4: Real-world deployment on Fetch robot. (a) Physical robot execution: given start and goal images (top row), vRRT plans a collision-free trajectory executed on the real robot (bottom row frames). (b) Planning dynamics: exploitation samples (blue) follow visual gradients while exploration samples (red) maintain coverage, demonstrating effective sim-to-real transfer.

Robot	Method	Metric	Target distance bins (rad)					Avg.
Robot	Method	Metric	$0.5$	$1.0$	$1.5$	$2.0$	$2.5$	Avg.
Franka	Dr.Robot	SR	92.0	53.0	21.0	11.0	2.0	35.8
		Error	0.048	0.315	0.603	0.878	1.097	0.588
		PSNR	30.19	26.83	23.44	22.34	21.29	24.82
	Ours	SR	100.0	99.0	84.0	70.0	41.0	78.8
		Error	0.015	0.019	0.098	0.159	0.420	0.142

Table 2: Visual-goal pose reconstruction results. Success rate (SR, %), mean joint error (Error, rad), and PSNR across configuration-space distance bins. vRRT demonstrates substantially better performance across all bins in both success rate and joint accuracy. Notably, vRRT maintains high PSNR–measuring visual similarity, the primary objective in visual-goal planning–across distances where Dr.Robot [ 30 ] shows degradation.

Figure 5: Qualitative pose reconstruction results. Reconstruction comparison under self-occlusion (ground truth in insets). Under occlusion, Dr.Robot [ 30 ] produces configurations with plausible overall silhouettes but incorrect occluded joint angles, whereas vRRT recovers poses more consistent with complete robot structure through exploration-based disambiguation.

Method	J1	J2	J3	J4	J5	J6	J7	Avg.
RoboPEPP [ 10 ]	0.052	0.072	0.068	0.206	0.125	0.579	–	0.184
HoRoPose [ 2 ]	0.193	0.123	0.197	0.054	0.085	0.071	0.355	0.154
Dr.Robot [ 30 ]	0.077	0.040	0.106	0.064	0.104	0.137	0.617	0.164
Ours	0.088	0.030	0.095	0.030	0.162	0.082	0.094	0.083

Table 3: Per-joint reconstruction errors on Panda-3CAM-Azure dataset. Comparison of per-joint angular errors (rad) across feed-forward regression methods (RoboPEPP, HoRoPose), gradient-based optimization (Dr.Robot), and vRRT.

Noise Std.	Target distance bins (rad)					Avg.
Noise Std.	$0.5$	$1.0$	$1.5$	$2.0$	$2.5$	Avg.
0.05	90.7	92.8	91.0	93.0	93.8	92.3
0.10	89.2	91.0	91.2	91.3	90.2	90.6
0.15	86.8	90.0	91.0	89.7	90.0	89.5
0.20	87.8	89.5	90.5	88.3	89.0	89.0

Table 4: Robustness to noisy goal configurations. Motion planning success rates (%) when vRRT uses noisy goal estimates alongside visual objectives– simulating scenarios with degraded sensors or low-precision encoders. High success rates across noise levels demonstrate effective visual-configuration fusion.

Figure 6: Qualitative results on Panda-3CAM-Azure dataset. Rendered poses overlaid on goal images demonstrate improved reconstruction quality. vRRT achieves better accuracy on occluded joints compared to Dr.Robot [ 30 ]

Figure 7: Ablation on vRRT tree expansion. Success rates (%) across distance bins under different (a) exploration ratios $r$ and (b) frontier sampling ratios $\eta$ on UR5e for visual-goal motion planning. The best results are obtained at $r=0.3$ and $\eta=0.6$ – $0.8$ , indicating that planning requires a balance between goal-directed expansion and exploration. Uniform sampling ( $\eta=0.0$ ) and excessive exploration ( $r=0.9$ ) both degrade performance, confirming the importance of visual guidance and adaptive prioritization.

Figure 8: Planning progression visualization. Planning progression at iterations T=1, 2, 3 for two scenarios. Blue: exploitation samples following visual gradients. Red: exploration samples for coverage. The coexistence of both strategies enables vRRT to discover feasible paths while avoiding local minima.

$\beta_{1}$	$\beta_{2}$	Target distance bins (rad)					Avg.
$\beta_{1}$	$\beta_{2}$	$0.5$	$1.0$	$1.5$	$2.0$	$2.5$	Avg.
0.5	0.9	38.7	29.4	26.9	25.7	27.5	29.6
0.7	0.9	76.5	68.0	66.3	59.0	49.0	63.8
0.9	0.9	87.3	87.5	85.2	76.2	62.7	79.8
0.99	0.9	78.0	70.7	67.6	61.7	47.8	65.2
0.9	0.99	87.0	88.3	83.7	77.5	58.0	78.9

Table 5: Effect of Adam moments parameters. Success rates (%) with varying

\beta_{1}

and

\beta_{2}

in inertial gradient expansion, evaluated on the UR5e robot under the visual-goal motion planning. Optimal performance at standard Adam values (

\beta_{1}=0.9

\beta_{2}=0.9

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	Frontier Sampling	Target distance bins (rad)					Avg.
	Frontier Sampling	$0.5$	$1.0$	$1.5$	$2.0$	$2.5$	Avg.
(a)	Trunc. geometric ( $\kappa$ =0.9)	87.3	87.5	85.2	76.2	62.7	79.8
(b)	Trunc. geometric ( $\kappa$ =0.7)	86.7	88.3	83.4	74.0	55.0	77.5
(c)	Trunc. geometric ( $\kappa$ =0.5)	86.3	85.7	82.6	72.5	54.2	76.2
(d)	Uniform	17.3	3.5	1.3	0.3	0.8	4.7
(e)	Top-K	86.8	86.7	82.7	71.3	53.2	76.1

Appendix

Table S1 : Ablation on frontier sampling strategies. Success rates (%) across C-space distance bins for frontier node selection methods. Higher

\kappa

achieves better performance through stochastic diversity. Uniform sampling (d) and Top-K (e) show limitations compared to

\kappa\!=\!0.9

at larger distances (2.0 and 2.5 rad).

	Inherited Optimizer State	Target distance bins (rad)					Avg.
	Inherited Optimizer State	$0.5$	$1.0$	$1.5$	$2.0$	$2.5$	Avg.
(a)	Naive GD (Eq. 1)	21.8	3.8	2.2	4.7	3.5	7.2
(b)	Momentum ( $\mu=0.5$ )	37.8	16.3	10.1	12.8	10.8	17.6
(c)	Momentum ( $\mu=0.7$ )	54.0	28.6	21.3	22.2	16.7	28.5
(d)	Momentum ( $\mu=0.8$ )	68.8	38.6	32.9	26.7	15.3	36.5
(e)	Momentum ( $\mu=0.85$ )	73.2	49.4	38.7	31.5	22.0	43.0

Appendix

Table S2 : Ablation on optimizer state inheritance. Success rates (%) across C-space distances for different inherited optimizer states in vRRT. Naive gradient descent without any optimizer state (a) is less effective at exploiting visual gradients. Inheriting a simple momentum term (b)–(g) substantially improves performance over stateless tree expansion. Adaptive optimizers that accumulate gradient statistics (h)–(k) further outperform the best momentum-only baseline, with Adam achieving the highest average success rate.

Robot	RRT ∗	Target distance bins (rad)					Avg.
Robot	RRT ∗	$0.5$	$1.0$	$1.5$	$2.0$	$2.5$	Avg.
Franka	w/ Dr.Robot goal	0.50	0.99	1.40	1.90	2.49	1.46
Franka	w/ vRRT goal	0.50	1.00	1.55	2.08	2.63	1.55
UR5e	w/ Dr.Robot goal	0.50	1.02	1.52	2.00	2.55	1.52
UR5e	w/ vRRT goal	0.50	1.01	1.55	2.10	2.59	1.55
Fetch	w/ Dr.Robot goal	0.47	0.98	1.49	1.93	2.46	1.47

Appendix

Table S3 : Comparing RRT ∗ path lengths across recovered goals. RRT ∗ paths to vRRT-recovered goals are consistently longer, suggesting the methods solve different problem subsets: vRRT addresses more challenging instances requiring longer paths, while Dr.Robot [ 30 ] succeeds on shorter-distance cases.

Figure S1 : Path comparison between vRRT and RRT ∗ . Paths from vRRT (yellow) and RRT ∗ (red) across three robots in six scenes each. vRRT produces paths similar to RRT ∗ despite using only visual objectives. Videos are provided in the project page.

Figure S2 : Fetch deployment across three scenes. Representative executions from three different obstacle configurations. Top row shows executed paths reaching visual goals. Bottom row shows planning dynamics where exploitation samples (blue) follow visual gradients while exploration samples (red) maintain coverage. The videos are provided in the project page.

Figure S3 : Visual-goal motion planning with generated goal images. To demonstrate broader applicability, we generate goals by prompting an generation model [ 6 ] to inpaint a Franka robot into Fetch scenes while maintaining background. Four tasks show paths matching synthesized poses, demonstrating potential to bridge generative models with executable planning.

	Scene 1	Scene 2	Scene 3	Total
Tasks	25	25	25	75
Successes	18	24	18	60

Appendix

Table S4 : Real-world Fetch experiments. Success counts across three scenes with 25 tasks each. vRRT achieves 80% success rate (60/75) in real-world deployment, demonstrating potential for sim-to-real transfer in visual-goal motion planning.

Figure S4 : Tree structure with frontier node visualization. Configuration space projected onto 2D via PCA for three planning instances (a-c). Frontier nodes (orange) represent configurations with low visual loss selected for effective expansion, while gray nodes indicate non-frontier configurations. The spatial distribution shows frontier nodes progressively concentrate toward the goal region, validating that our frontier-based sampling effectively prioritizes visually promising areas for exploration and exploitation.

Figure S5 : Tree structure colored by inertial gradient expansion steps. Robot joint configurations visualized in 2D via PCA across three planning instances (a-c). Node colors indicate inertial gradient tree expansion steps $i$ : low values (purple) represent nodes primarily reached through exploration, while high values (yellow) indicate nodes refined through extensive gradient optimization. Progressive color transitions show that state inheritance enables continuous optimization across tree branches. Notably, convergence patterns vary across instances: (a) exhibits a cohesive gradient flow where optimization steps increase smoothly toward the goal, whereas (b) and (c) show multiple distinct optimization branches reaching the target.

Robot (# of Gaussians)	Rendering	Frontier sampling	RRT operations	Collision checking
Franka (52,393)	80.7	2.6	11.9	4.7
UR5e (16,327)	60.6	4.0	21.6	13.8
Fetch (70,335)	84.7	1.9	9.4	4.1

Appendix

Table S5 : Computational cost per iteration. Average time allocation (%) when expanding 32 nodes per iteration across three robot platforms. Rendering dominates the cost indicating that reducing Gaussians directly improves vRRT efficiency.

Figure S6 : Representative failure cases. Target pose images (left) visually resemble predictions (right) but differ in joint space. Fetch and Franka: symmetric poses produce identical silhouettes. UR5e: self-occlusion allows hidden joints to rotate without visual change. These illustrate the inherent difficulty of visual-goal planning.

Motion Planning

Motion planning finds safe, feasible paths and trajectories for robots and vehicles amid geometric, dynamic, and task constraints.

See all 405 papers

Oct 26, 2026

Reward-Guided RRT* for Budgeted Task and Motion Planning

Robotic missions involving spatially distributed tasks often require jointly selecting rewarding locations and planning feasible motions under limited travel budgets. This paper proposes Reward-Guided RRT*, a new sampling-based framework that constructs a budget-feasible mission through a sequence of locally joint task-selection-and-motion-planning stages, evaluated directly in the continuous state space, without requiring a precomputed task-transition graph. The method incrementally builds a mission using reward-guided RRT* searches, while a return-feasibility procedure ensures that each selected task can be visited without violating the mission budget. Experiments with increasing obstacle complexity and number of tasks show solutions comparable to VNS-PRM* while requiring substantially fewer sampled states. We also extend the framework to Dubins steering and evaluate it in CoppeliaSim with a scaled car-like robotic platform, demonstrating its ability to incorporate vehicle kinematic constraints.

Vinícius Araújo de Oliveira, Gabriel Ferreira Mendes Lourenço, Paulo L. J. Drews-Jr +2

Computer Vision and Robotics Laboratory (VeRLab), Dept. of Computer Science, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil. · NAUTEC, Center for Computational Science, Universidade Federal do Rio Grande - FURG, Rio Grande, Brazil. · Dept. of Electronics Engineering, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil.

Motion Planning Sampling-Based Motion Planning Constrained Optimization

Ask PDF

May 6, 2024

Greedy Heuristics for Sampling-Based Motion Planning in High-Dimensional State Spaces

Informed sampling techniques accelerate the convergence of sampling-based motion planners by biasing sampling toward regions of the state space that are most likely to yield better solutions. However, when the current solution path contains redundant or tortuous segments, the resulting informed subset may remain unnecessarily large, slowing convergence. Our prior work addressed this issue by introducing the greedy informed set, which reduces the sampling region based on the maximum heuristic cost along the current solution path. In this article, we formally characterize the behavior of the greedy informed set within Rapidly-exploring Random Tree (RRT*)-like planners and analyze how greedy sampling affects exploration and asymptotic optimality. We then present Greedy RRT* (G-RRT*), a bi-directional anytime variant of RRT* that leverages the greedy informed set to focus sampling in the most promising regions of the search space. Experiments on abstract planning benchmarks, manipulation tasks from the MotionBenchMaker dataset, and a dual-arm Barrett WAM problem demonstrate that G-RRT* rapidly finds initial solutions and converges asymptotically to optimal paths, outperforming state-of-the-art sampling-based planners.

Phone Thiha Kyaw, Anh Vu Le, Rajesh Elara Mohan +1

Space & Terrestrial Autonomous Robotic Systems (STARS) Laboratory, University of Toronto Institute for Aerospace Studies, 4925 Dufferin Sreet, Toronto, M3H 5T6, Ontario, Canada. · Advanced Intelligent Technology Research Group, Faculty of Electrical and Electronics Engineering, Ton Duc Thang University, Ho Chi Minh City, 700000, Vietnam. · ROAR Lab, Engineering Product Development, Singapore University of Technology and Design, Singapore, 487372, Singapore.

Sampling-Based Motion Planning Tree Search

Ask PDF

Oct 18, 2026

Dream What I Can: Sparse Visual Planning for Long-Horizon Robot Manipulation

Video models provide rich priors for generalizable robotic manipulation, but plausible visual futures are not always executable. We introduce DreamCan, a hierarchical policy that grounds video models through sparse visual planning: a high-level planner adapted from a pre-trained video model and a low-level visuomotor policy. Rather than predicting dense trajectories, the high-level planner generates sparse visual subgoals that balance reachability and task utility. From offline, action-free robot videos, we estimate these quantities and use dynamic programming to select subgoal sequences without manual annotations. The resulting subgoals are more reliably attained than uniformly spaced alternatives, and the full system outperforms language, dense-video, and uniform-subgoal planners, reaching high planning success rates on Language-Table tasks and 91.2% success on LIBERO-Long with a 0.1B-parameter low-level policy. Additional suboptimal videos and test-time compute further improve success. The system also performs challenging real-world bimanual tasks, including box folding and packing with unseen object combinations. More results are available at https://dreamcan.github.io/.

Chongkai Gao, Wenlong Huang, Haonan Chen +5

Stanford University · National University of Singapore · Harvard University +1

Long-Horizon Robotic Manipulation Robotic Manipulation Robot Learning from Video

Ask PDF

Abstract

Motion Planning

Explore similar work

Reward-Guided RRT* for Budgeted Task and Motion Planning

Greedy Heuristics for Sampling-Based Motion Planning in High-Dimensional State Spaces

Dream What I Can: Sparse Visual Planning for Long-Horizon Robot Manipulation