PlantShade: Predicting Plant Shadows for Lighting-Aware Robotic Agricultural Operation

Longchao Da; Xiaoou Liu; Xingjian Li; Lirong Xiang; Hua Wei

cs.ROSep 17, 2026

PlantShade: Predicting Plant Shadows for Lighting-Aware Robotic Agricultural Operation

Authors: Longchao Da, Xiaoou Liu, Xingjian Li, Lirong Xiang, Hua Wei

Organizations: School of Computing and Augmented Intelligence, Arizona State University · Automation and Robotics Laboratory, Department of Biological and Environmental Engineering, Cornell University

Abstract

Plant growth and agricultural production form the foundation of a country's sustainable development and directly impact human livelihoods. Recent advances in frontier artificial intelligence have enabled scientific agriculture with strong potential to improve crop productivity. In this paper, we identify the importance and inherent complexity of plant shade simulation, as shading is a critical factor influencing plant growth. To advance this field and promote broader societal benefits, we focus on two main contributions. First, we introduce a comprehensive plant growth and shade dataset covering four plant species, including soybean, tomato, sugarbeet, and strawberry. The dataset includes top-down viewpoints with a supplementary light along a circular trajectory, casting dynamic shadows across multiple growth stages and diverse observation complexities. Second, we propose generative shade simulation based on diffusion models, enabling realistic shade generation for unseen plants and supporting downstream robotic tasks such as perception, lighting control, and view planning. The model incorporates temporal conditioning to facilitate flexible shade simulation across different time stages. We conduct both quantitative and qualitative evaluations to assess model performance. This work provides a foundational study for plant-aware shade modeling and has meaningful implications for broader agricultural and robotic applications.

Figures & tables

Fig. 1: Top: The PlantShade system captures plant images under a controlled robotic lighting setup, where a movable supplemental light follows a circular trajectory to generate shadow variations. The dataset is used to train a light-conditioned generative model for shadow prediction. Given a plant image and light-position prompts, the model produces a shadow map. Bottom: The predicted shadow enables applications including lighting optimization through placement-adjust, light exposure analysis via heatmap estimation, and shade-aware robotic pruning by simulating branch removal and assessing impacts on self and mutual-shading.

Panel of Figure 2 — Fig. 2: Simulation framework and data generation. Top: Data generation setup and flowchart with sugar beet in the sample. Bottom: Zoomed-in sample images for strawberry and soybean across growth stages.

Parameter	Specification
Total Image Pairs	38,500
Resolution	$1920\times 1080$
Camera Height	1.8 m (Nadir View)
Plant Species	4 ( Tomato , Soybean , Sugar beet , Strawberry )
Plant Age Range	7 – 119 days
Layout Configurations	$1\times 1$ , $1\times 3$ , $3\times 5$

TABLE I: Key specifications of the PlantShade dataset.

Panel of Figure 4 — Fig. 3: UE5 and Helios comparison for a sunlit tomato canopy. Left: Differences in visual appearance between renderings produced by Helios’s radiation module and UE5, with the background removed for canopy shade clarity. Right: Sunlit canopy coverage under a rotating supplementary light source, demonstrating similar illumination trends between Helios (computed from all leaves) and UE5 (estimated from observed leaves only).

Fig. 4: Architecture of the PlantShade shadow generative model. The framework combines a text-encoded supplementary light position with structural guidance from RGB and edge features to condition a diffusion backbone, generating shadow maps consistent with plant geometry and lighting configuration.

		Single				$1\times 3$				$3\times 5$
Metric	Method	Tomato	Soybean	Strawberry	Sugar beet*	Tomato	Sugar beet	Soybean	Strawberry	Tomato	Sugar beet	Soybean*	Strawberry*
mIoU $\uparrow$	PlantShade	0.2636	0.4371	0.3413	0.0314	0.2561	0.4088	0.3574	0.3970	0.3283	0.5530	0.0863	0.1094
mIoU $\uparrow$	Diffusion	0.0080	0.0098	0.0033	0.0089	0.0099	0.0082	0.0191	0.0095	0.0292	0.0349	0.0415	0.0441
B-IoU $\uparrow$	PlantShade	0.3808	0.3921	0.3911	0.0451	0.3661	0.3730	0.3390	0.3587	0.3996	0.4401	0.1238	0.1234
B-IoU $\uparrow$	Diffusion	0.0176	0.0158	0.0073	0.0119	0.0235	0.0143	0.0309	0.0162	0.0550	0.0481	0.0554	0.0564
LPIPS $\downarrow$	PlantShade	0.0660	0.0638	0.0428	0.3713	0.0925	0.0671	0.1251	0.0848	0.2936	0.2119	0.6464	0.6575

TABLE II: Quantitative comparison between PlantShade and Stable Diffusion under different layout configurations. Purple columns marked with * indicate out-of-domain (OOD) settings.

Fig. 5: Performance from in-distribution (ID) to OOD settings. (1) MSE comparison under both Single and $3\times 5$ configurations shows that both PlantShade and Diffusion perform worse in OOD scenarios, indicating that OOD plant placement is consistently more challenging. (2) However, the Radar comparison on the $3\times 5$ OOD setting still demonstrates that PlantShade consistently outperforms the Diffusion baseline across four metrics, highlighting stronger robustness under distribution shift.

Fig. 6: Overview of the PlantShade dataset and representative generation results. The left circular diagram summarizes the four core dimensions of the dataset: growth cycle coverage, plant species diversity, display configurations, and supplemental light sampling with 100 circular positions. The top-left panel shows representative samples of the four plant species included in the dataset. The top-right panel presents shadow estimation results across different growth stages, illustrating temporal consistency in light-driven shadow formation. The bottom-right panel shows generated shadow maps under varying spatial placement configurations, highlighting the model’s ability to generalize across density and layout complexity. The light sampling is shown in Fig. 2 .

Fig. 7: Overview of the photosynthetic gain estimation pipeline and results. Left: Depth and predicted shadow maps are used to estimate canopy geometry and shaded fraction. Middle: A species-specific light–response model converts shading into canopy photosynthetic rates. Right: Comparison of No Supplemental Light, Optimal Placement (GT), and Optimal Placement (Prediction). Bottom: Results for soybean, strawberry, sugar beet, and tomato under single, $1\times 3$ , and $3\times 5$ layouts, showing consistent photosynthetic gains with optimal supplemental lighting.

Synthetic Data Generation for Robotics

Synthetic data generation for robotics creates simulated scenes, sensor readings, and action trajectories to train and evaluate robots.

See all 303 papers

AgriGen: Large-Scale Scene Generation Framework for Photorealistic Agricultural Robotics Simulation
Sep 22, 2026Utkarsh Bajpai, Serge Tleiji, Cédric Pradalier +1Robotics Simulation Synthetic Data Generation for Robotics
Georgia Tech-CNRS IRL2958, France · Notre Dame University, Lebanon; work done during internship at 1
LettuceVisSim: A Simulator That Generates Lettuce Image Time-series for Vision-Based Reinforcement Learning
Sep 11, 2026Ziye Zhu, Bert van 't Ooster, Congcong Sun +2Synthetic Data Generation Image Generation
Agricultural Biosystems Engineering Group, Wageningen University & Research, The Netherlands · Biometris, Wageningen University & Research, The Netherlands
Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning
Mar 9, 2026Heesup Yun, Isaac Kazuo Uyehara, Earl Ranario +4VLM Evaluation Agricultural Image Analysis
University of California, Davis

Sep 22, 2026J/K move · Enter open · S save

AgriGen: Large-Scale Scene Generation Framework for Photorealistic Agricultural Robotics Simulation

Utkarsh Bajpai, Serge Tleiji, Cédric Pradalier, Stéphanie Aravecchia

Georgia Tech-CNRS IRL2958, France · Notre Dame University, Lebanon; work done during internship at 1

Robotics Simulation Synthetic Data Generation for Robotics 3D Scene Generation Agricultural Robotics

Open paper AskPDF arXiv

Agricultural robotics is advancing rapidly, yet progress remains constrained by limited field access, lack of control over field conditions, geographic variability, and seasonal crop cycles. These factors make it difficult and costly to acquire diverse agricultural datasets, resulting in limited evaluation and reduced system robustness. While other robotics domains have scaled learning and evaluation through high-fidelity simulation, agricultural robotics still lacks comparably capable tools. In this paper, we present a ROS-integrated framework, built on Isaac Sim, for large-scale procedural generation of agricultural environments. The framework supports photorealistic rendering, physics simulation, and domain randomization at scales relevant to robotics research, with built-in support for row crops, orchards, and vineyards and straightforward extensibility to additional crop categories. Project Page: https://baj31415.github.io/agrigen/

Abstract

Synthetic Data Generation for Robotics

Explore similar work

AgriGen: Large-Scale Scene Generation Framework for Photorealistic Agricultural Robotics Simulation

LettuceVisSim: A Simulator That Generates Lettuce Image Time-series for Vision-Based Reinforcement Learning

Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning