A Vision-Language Model (VLM)-based Pipeline for End-to-End Procedural Modeling of Field-Grown Maize from Point Clouds

Mozhgan Hadadi; Talukder Z. Jubery; Adarsh Krishnamurthy; Baskar Ganapathysubramanian

cs.CVOct 2, 2026

A Vision-Language Model (VLM)-based Pipeline for End-to-End Procedural Modeling of Field-Grown Maize from Point Clouds

Authors: Mozhgan Hadadi, Talukder Z. Jubery, Adarsh Krishnamurthy, Baskar Ganapathysubramanian

Abstract

Editable 3D models of field-grown crops support high-throughput phenotyping and in silico breeding trials, but building them from scanned point clouds requires organ-level segmentation and fitting. Procedural generators can turn an organ-level parameter set into an analysis-suitable 3D model, but obtaining that set requires hours of manual tuning per plant or segmentation models trained on species-specific labels. We present an automated pipeline that reconstructs procedural maize models from raw 3D point clouds without manual tuning or species-specific training data. A multimodal vision-language model (VLM) annotates leaf midlines in rendered orthographic views. Deterministic geometric algorithms back-project the annotations onto the point cloud, merge them into 3D leaves by cross-view consensus, and grow the midlines to full blades on an orientation-weighted surface graph. Measured organ parameters populate a plant descriptor for a Non-Uniform Rational B-Spline (NURBS)-based procedural model generator. Each leaf surface is then refined against its scan points by differentiable NURBS fitting. The pipeline reached a median whole-plant Chamfer distance of 5.4 mm on 100 genotypically diverse field-grown maize plants from the MaizeField3D dataset. The reconstructions were closer to the scans than those of an earlier semi-automated pipeline based on manual annotations. The pipeline recovered 1,017 of 1,023 (99.4%) curated reference leaves at an intersection-over-union of at least 0.5 without using those labels as input. These results show that VLM annotations become usable organ-level measurements when downstream geometric stages can correct them. This makes automated generation of editable 3D plant assets feasible at the scale of modern phenotyping experiments.

Figures & tables

Figure 1: The nine pipeline stages (preprocess, render, VLM, masks, consensus, blade-level segmentation and node repair, parameters plus procedural plant generation (PPG), NURBS-Diff, and report), read along each row and turning back at the end of each row. Each stage is shown beside its own output for one of the evaluated plants (Kui2021). Rendering, annotation, masking, and consensus share one ordered set of views. Sparse per-view evidence is expanded to full blades, and misassigned blade material is corrected before parameters are measured. Node positions, leaf lengths, B-spline midline control points, width profiles, azimuths, pitches, and stalk diameters are measured from the corrected blade clouds and reconstructed stalk. These values automatically populate the FloraForge plant descriptor, from which the initial procedural model is generated. NURBS-Diff then uses gradient-based optimization to fit each attached blade surface more closely to its segmented points.

#	Genotype	Leaves	#	Genotype	Leaves
1	M162W	15	6	4226	12
2	CML69	15	7	78551S	8
3	Kui2021	12	8	A679	8
4	CML238	12	9	CI90C	10
5	CML331	12	10	T8	14

Table 1: The ten plants reported leaf by leaf, identified by genotype, with the number of leaves recovered by the pipeline. All scans are from the MaizeField3D release [ Kimara et al., 2026 ] . These ten plants account for

118

of the

1{,}072

leaves reconstructed across the

100

plants.

Panel of Figure 2 — Figure 2: The ten plants reported leaf by leaf ( Table 1 ), shown as the orthographic front view of each canonicalized point cloud in the colors recorded by the scanner. Each cloud is the input from which the pipeline of Section 2 reconstructs a procedural model.

Panel of Figure 3 — Figure 3: The eight orthographic renderings of genotype Kui2021 used as input to the VLM stage. The view set covers the front, back, and six side viewpoints, so a leaf that is self-occluded in one view is generally visible in several others. The panels are cropped to the plant. The full renders also include the three corner overlays of Section 2.3 .

Panel of Figure 4 — Figure 4: The annotations returned for Kui2021, one per rendered view of Figure 3 . Panels (a) to (h) show the leaf pass of Prompt 1, with one uniquely colored midline per visible leaf and magenta dots at the stalk-leaf junctions. Panel (i) shows the separate stalk pass of Prompt 2, which is applied to the front view only and returns the red stalk stroke and the two magenta endpoint squares used to reconstruct the stalk centerline ( Section 2.7 ). The mask stage reassigns the colors, and consensus reconciles the view-local stroke identities and counts across views ( Section 2.6 ). In this example, the leaf count differs between views, a disagreement that consensus resolves.

Panel of Figure 5 — Figure 5: From per-view masks to a single leaf set in 3D, on Kui2021. (a) The mask stage applied to all rendered views. Each annotation stroke is skeletonized, cut at its junctions, regrouped by tangent continuation, and repainted with one color per view-local leaf. This regrouping can separate two leaves drawn as one stroke and assign disconnected fragments to one leaf identity. The mask sheets are labeled by view, in the order listed in Section 2.3 . The number of masks differs between views because a leaf occluded in one view is visible in another. The colors are view-local identifiers and do not correspond between views. (b) The masks of all eight views, back-projected onto the point cloud with the camera that rendered them and reconciled by cross-view consensus into one numbered leaf set, drawn as one midline curve per leaf over the cloud (gray). Only in this panel do the colors identify the same physical leaf across views, as established by the consensus rule of Section 2.6 .

Figure 6: Ellipsoid-envelope diagnostic and correction of a misassigned blade fragment, on CML331. (a) A blade is tiled by a chain of ellipsoids, one per patch of equal arc length $\Delta s$ , each with an axial half-length $\ell_{i}$ , a half-width $a_{i}$ measured across the blade, and a half-thickness $b_{i}$ through it. The score of Eq. 2 divides a point’s offset $\mathbf{d}$ from the patch center by these half-axes. Thus $\sigma=1$ is the boundary of a padded ellipsoid, and $\sigma\leq 1.5$ is the slack allowed by the acceptance test. (b) Leaf $9$ after the initial assignment, drawn in the plant’s front view with the rest of the plant in gray and with points colored by the score of their own leaf’s envelope. The blade sits at a median score of $\sigma=1.3$ , while the part of the fragment lying on this leaf reaches $\sigma=42$ , placing those points far outside this leaf’s envelope. The correction moved $90$ points in total, taken from this leaf and from leaf $7$ . (c) The same points after the accepted move , scored against leaf $11$ ’s envelope. Here the fragment sits at $1.1$ , within leaf $11$ ’s acceptance envelope.

Panel of Figure 7 — Figure 7: End-to-end reconstruction of five of the $100$ evaluated plants, with the genotype named above each column. From top to bottom, the rows show the input scan in the colors recorded by the scanner, the front-view leaf annotation used for reconstruction (one of the eight leaf-view annotations), the blade-level segmentation of Section 2.7 , the initial procedural model generated by FloraForge [ Hadadi et al., 2026 ] from the measured plant descriptor, and the same procedural model after NURBS-Diff refinement of its leaf surfaces against the segmented blade points. Each row is labeled with the stage of Figure 1 that produced it. In the segmentation row, each leaf is grown from its consensus midline out to the full blade, and ambiguous points are left unlabeled (gray). All rows except the annotation are rendered with the same camera within a column, so a leaf can be followed from the scan to the refined surface.

Genotype	Leaves	Front	Mean	Fewest	All eight	Two only	Components
M162W	15	14	$13.0$	11	9	1	109
CML69	15	14	$13.2$	11	11	0	113
Kui2021	12	11	$10.5$	9	7	1	85
CML238	12	11	$11.0$	10	9	0	97
CML331	11	11	$9.9$	9	8	0	84
4226	12	12	$11.4$	10	9	0	96

Table 2: Per-view visibility of the recovered leaves. “Leaves” is the number of global leaves established by cross-view consensus. “Front” is the number of these leaves resolved by the front view alone. “Mean” and “Fewest” give the mean and the minimum of this count over the eight views. “All eight” counts the leaves resolved in all eight views, and “Two only” counts the leaves resolved in exactly two views, i.e., at the two-view support floor. “Components” is the total number of per-view leaf components produced by the mask stage over the eight views.

Outcome	Leaves	Conditions met	Leaves affected
Existing node preserved	114	114	n/a
Unsafe node repaired by projection	1	1	T8 L11
Projected because required	2	1	CML331 L8, L12
Unresolved	1	0	CML69 L14
Total	118	116

Table 3: Outcome of the constrained node rule for final node placement over the

118

leaves of the ten plants. The two conditions require a base-to-node gap of at most

6

cm and a spacing of at least

4

cm to the nearest node. “Conditions met” is the number of leaves in each group whose final node satisfies both. Two leaves fail the base-to-node gap condition while satisfying the spacing condition.

Metric	Value (mm)
Median RMSE	$1.68$
Mean RMSE	$2.17$
Std. dev.	$1.73$
Interquartile range	$1.10$ to $2.79$
90th percentile	$3.93$
Max	$13.10$

Table 4: B-spline midline reconstruction RMSE across the

118

reconstructed leaves. Each leaf midline is represented by 8 control points with a degree-3 interpolating B-spline, and the residual is measured against the geodesic midline path from which the control points were sampled.

Leaf	Final RMS (mm)	Blade points
L1	$4.78$	615
L2	$5.89$	931
L3	$7.60$	1030
L4	$8.24$	1161
L5	$5.95$	1113
L6	$5.39$	1093

Table 5: Per-leaf distance from the observed blade points to the fitted surface for genotype 78551S on the final selected geometry. Leaves are numbered bottom-to-top. Values are the exact point-to-surface distance of Section 3.3 , in millimeters. Table 1 gives a per-plant summary.

Figure 8: Fidelity of the final refined surfaces at two scales. (a) Whole-plant symmetric Chamfer distance between the input point cloud and the assembled full-plant mesh for the ten genotypes of Table 1 . The dashed line marks the median. (b) Blade-level residual between each final fitted surface and its observed blade points on Kui2021, computed as the exact point-to-surface distance of Section 3.3 , with leaves numbered bottom-to-top.

Panel of Figure 9 — Figure 9: Agreement between traits measured on the exported surface and on the segmented blade cloud, over the $118$ leaves of the ten genotypes of Table 1 . 9(a) Blade length and 9(b) node height against the $1{:}1$ line, annotated with mean absolute error, coefficient of determination, and leaf count. 9(c) Vertical insertion angle and 9(d) azimuth, as the per-leaf cosine of the angular disagreement, which is bounded above by one and remains well defined across the $0/360^{\circ}$ wrap. Bars are grouped by genotype, and the dashed line marks exact agreement.

Figure 10: The pipeline over all

100

evaluated plants, on the final refined geometry. (a) Whole-plant symmetric Chamfer distance against the input cloud, one entry per plant; the dashed line marks the median. (b) Blade-level residual over all

1{,}072

reconstructed leaves, measured as the exact point-to-surface distance of Section 3.3 . Three leaves lie beyond the axis. (c) Leaves recovered per plant.

Figure 11: Whole-plant fidelity of the proposed pipeline against the PSO+NURBS-Diff pipeline of [ Hadadi et al., 2025 ] on the same

100

plants, both recomputed under one protocol. (a) One point per plant. Points below the dashed line (

99

of the

100

) are plants that this work reconstructs closer to their scan. (b) The two distributions with their medians.

Metric	PSO+NURBS-Diff [ Hadadi et al., 2025 ]	VLM+NURBS-Diff (this work)	Ratio
Surface initialization	PSO over $32$ to $33$ variables	Measured plant descriptor	n/a
Organ segmentation	Manual / semi-automated	Reviewed VLM annotations plus geometric processing	n/a
Whole-plant Chamfer distance, median (mm)	$9.2$	$\mathbf{5.4}$	$1.7\times$
plants with the lower distance	$1$ of $100$	$\mathbf{99}$ of $\mathbf{100}$	n/a
Per-plant geometric-core compute (s)	$3430\pm 490$	$601\pm 55$	n/a
of which initialization (s)	$2740\pm 180$ (PSO)	$82\pm 24$ (measured descriptor)	$33\times$

Table 6: Comparison of the proposed pipeline against the PSO+NURBS-Diff pipeline of [ Hadadi et al., 2025 ] . Time values are the mean

\pm

standard deviation of per-plant times over the ten plants of Section 2.1 for the proposed pipeline, and the values published in [ Hadadi et al., 2025 ] for the previous one. No ratio is given on the summed compute row, since our

601

s includes

382

s of segmentation with no counterpart in the

3430

s. The two indented rows below it compare stages of matching scope. Historical uncertainties are reproduced as published. The two fidelity rows are recomputed here for both pipelines under one protocol over the same

100

plants, as described in the text.

Stage	Time per plant (s)
VLM annotation	$214$
Mask extraction and leaf separation (8 views)	$187\pm 21$
Cross-view consensus	$47\pm 3$
Blade segmentation and node policy	$148\pm 18$
Parameter measurement and procedural assembly	$82\pm 24$
NURBS-Diff refinement (optimization only)	$137\pm 38$

Table 7: Per-plant runtime over the ten plants of Section 2.1 , reported as mean

\pm

standard deviation. The annotation row is elapsed wall-clock time for eight leaf-view tasks and one stalk task per plant, pooled over

19

plants annotated in batches. The annotation row is given without a deviation because the batches give totals and not per-plant times, and the batch means span

209

233

s. Preprocessing, rendering, assembly, and reporting were not timed.

Genotype	Manual	Consensus	Exported	Difference
M162W	15	15	15	0
CML69	16	15	15	1
Kui2021	13	12	12	1
CML238	13	12	12	1
CML331	13	11	12	1
4226	12	12	12	0

Table 8: Recovered leaves against a manual count of the same scans. “Manual” is the number of leaves counted by hand from the rendered views. “Consensus” is the number established by cross-view consensus and “Exported” the number that reached a refined surface. The two differ on CML331, where the fragment correction of Section 2.7 recovered a twelfth leaf that consensus had not established. “Difference” is the manual count less the exported count.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1: The two prompts used in the VLM stage ( Section 2.4 ). Prompt 1 is applied to each of the eight rendered views and produces the leaf midlines and stalk-node dots shown in Figure 4 . Prompt 2 is applied to the front view only and produces the red stalk stroke and the two magenta endpoint squares used to reconstruct the 3D stalk centerline ( Section 2.7 ).

Stage	Role	Value	Unit	Src
Preprocessing	Unit conversion applied to all input clouds	$100$	cm m -1	M
	Radial percentile defining the stalk-base column	$20$	%	M
Rendering	Metric scale bar drawn in each view	$10$	cm	M
Annotation	Aspect-ratio tolerance accepting a returned canvas	$0.05$	n/a	M
	Silhouette IoU floor for a leaf annotation	$0.60$	n/a	M
	Silhouette IoU floor for a stalk annotation	$0.40$	n/a	M

Appendix

Table 1: Numeric parameters of the pipeline, grouped by the stage that applies them. The Src column gives the provenance code ( P , L , A , or M ) defined in the text. In the Unit column, “n/a” indicates that no physical unit applies.

Panel of Table 1 — Table 1: Per-plant qualitative outputs for the ten maize plants reported leaf by leaf, in the genotype order of Table 1 . The pipeline stage from Figure 1 is listed below each column heading. Each row shows the canonicalized input point cloud, the consensus leaf midlines, the blade-level segmentation after geometric correction, the initial procedural model generated from the measured descriptor, and the procedural model after NURBS-Diff refinement. Panels in each row use the same camera. Each plant is framed separately, so apparent size is not comparable across rows.

Genotype	Leaves	Blade RMS (mm)	Plant CD (mm)	Rot.	Fold
M162W	15	$5.8$	$6.7$	3	3
CML69	15	$5.5$	$7.7$	4	3
Kui2021	12	$6.0$	$9.7$	0	0
CML238	12	$5.0$	$8.3$	1	0
CML331	12	$6.1$	$8.8$	0	3
4226	12	$4.5$	$5.9$	1	2

Appendix

Table 1: Per-plant blade-level and whole-plant fidelity, with counts of adopted geometric modifications. Blade RMS uses the exact point-to-surface measure of Section 3.3 , and Plant CD is computed against the input cloud. “Rot.” denotes the gated rigid rotation, and “Fold” denotes the mid-blade fold repair. The final row gives totals for the counts and medians for the distances.

Procedural 3D Generation

See all 10 papers

Mar 23, 2026

A Vision Language Model for Generating Procedural Plant Architecture Representations from Simulated Images

Three-dimensional (3D) procedural plant architecture models have emerged as an important tool for simulation-based studies of plant structure and function, extracting plant architectural parameters from field measurements, and for generating realistic plants in computer graphics. However, measuring the architectural parameters and nested structures for these models at the field scales remains prohibitively labor-intensive. We present a novel algorithm that generates a 3D plant architecture from an image, creating a functional structural plant model that reflects organ-level geometric and topological parameters and provides a more comprehensive representation of the plant's architecture. Instead of using 3D sensors or processing multi-view images with computer vision to obtain the 3D structure of plants, we proposed a method that generates token sequences that encode a procedural definition of plant architecture. This work used only synthetic images for training and testing, with exact architectural parameters known, allowing testing of the hypothesis that organ-level architectural parameters could be extracted from image data using a vision-language model (VLM). A synthetic dataset of cowpea plant images was generated using the Helios 3D plant simulator, with the detailed plant architecture encoded in XML files. We developed a plant architecture tokenizer for the XML file defining plant architecture, converting it into a token sequence that a language model can predict. The model achieved a token F1 score of 0.73 during teacher-forced training. Evaluation of the model was performed through autoregressive generation, achieving a BLEU-4 score of 94.00% and a ROUGE-L score of 0.5182. This led to the conclusion that such plant architecture model generation and parameter extraction were possible from synthetic images; thus, future work will extend the approach to real imagery data.

Heesup Yun, Isaac Kazuo Uyehara, Ioannis Droutsas +4

Biological and Agricultural Engineering, One Shields Avenue, Davis, 95616, CA, USA · Department of Plant Sciences, One Shields Avenue, Davis, 95616, CA, USA · Wageningen University & Research, P.O. Box 9101, 6700 HB Wageningen, The Netherlands +1

Procedural Content Generation Plant Phenotyping 3D Reconstruction Vision-Language Models

Ask PDF

Sep 7, 2026

Zero-Shot 3D Plant Organ Segmentation with SAM3 and Semantic NeRFs

Accurate 3D plant organ segmentation is fundamental to automated phenotyping. Existing approaches rely on annotated training data or species-specific model configurations. We present an annotation-free pipeline for 3D plant organ segmentation, combining text-prompted SAM3 segmentation with semantic neural radiance fields (NeRFs). Given only multi-view RGB images and a list of class names, our zero-shot pipeline produces semantically labeled 3D point clouds without manual annotation, per-species fine-tuning, or domain-specific preprocessing. Multi-view NeRF fusion acts as effective implicit consensus mechanism that lifts imperfect per-frame masks into accurate 3D labels. On a controlled Begonia maculata testbed the SAM3 pipeline achieves 92.6% mIoU, reaching 95.9% of the oracle upper bound established with perfect ground-truth masks. The pipeline was further evaluated on a new dataset spanning ten diverse plant point clouds reaching an average 0.856 mIoU, with leaf and pot IoU above 0.91 and 0.90 for every species, respectively. These results demonstrate that annotation-free 3D plant organ segmentation is now feasible and approaching the range of supervised methods.

Andreas Gilson, Laura Hennig, Peter Pietrzyk

Fraunhofer Institute for Integrated Circuits (IIS), Fürth, Germany · Otto-Friedrich-Universität Bamberg, Germany · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany

Open-Vocabulary 3D Segmentation 3D Semantic Segmentation

Ask PDF

Sep 1, 2026

Automated Maize Ear Phenotyping Using 3D Reconstructions

Maize kernel traits such as row number, kernels per row, and kernel size vary largely for genetic reasons and are consistently associated with regions of the genome that influence yield. Manual measurement of these traits, however, cannot keep pace with the volume of maize generated in a breeding program. To address this, we developed and validated a fully automated pipeline for extracting these traits from 3D point clouds of corn ears, built on a recently developed video-to-point-cloud platform. Raw video frames are processed through COLMAP and NeRF, the ear is isolated via density-based separation, and the point cloud is distance-calibrated to physical units. The calibrated ear point cloud was Z-axis aligned via PCA and cylindrically unwrapped to a 2D image. We enhanced contrast and performed zero-fine-tuning instance segmentation using Cellpose-SAM. A triple-juxtaposed unwrap strategy was used to prevent double-counting at the seam. The pipeline achieved kernel count R^2 = 0.921 (MAPE = 10.33%) and kernel row number within +-2 rows for 95.2% of ears (MAE = 0.75 rows) on a 168-ear held-out set from the 268-ear labeled dataset. The resulting multi-trait dataset has known genotype identity for each ear, positioning it for phenotype-to-genotype association analyses.

Ritwesh A. Kumar, Som Tripathi, Peja Matthews +5

Department of Electrical and Computer Engineering, Iowa State University, Ames, IA 50011, USA · Translational AI Research and Education Center, Iowa State University, Ames, IA 50011, USA · Department of Mathematics and Computer Science, Fayetteville State University, Fayetteville, NC 28301, USA +3

Plant Phenotyping

Ask PDF

Abstract

Procedural 3D Generation

Explore similar work

A Vision Language Model for Generating Procedural Plant Architecture Representations from Simulated Images

Zero-Shot 3D Plant Organ Segmentation with SAM3 and Semantic NeRFs

Automated Maize Ear Phenotyping Using 3D Reconstructions