Field imaging framework for morphological characterization of aggregates with computer vision: Algorithms and applications

Haohang Huang

Figure 1.1 : Schematic outline of the dissertation

Figure 2.1 : Natural aggregate consumption in the United States (historical and projected) after Kelly (1998a)

Figure 2.2 : Value of Non-fuel Minerals Produced in 2020 ( Mineral, 2021a )

Figure 2.3 : Value of Crushed Stone Produced in 2020 ( Mineral, 2021a )

Figure 2.4 : Value of Construction Sand and Gravel Produced in 2020 ( Mineral, 2021a )

Figure 2.5 : Blasting procedure in the quarrying process. Source: Quarry (2017a)

Figure 2.6 : Rock breaking procedure for over-sized rock fragments. Source: Rammer Hammers (2013a)

Figure 2.7 : (a) Gyratory crusher, (b) jaw crusher, (c) impact crusher, and (d) cone crusher. Sources: Jankovic (2015a) ; BHS (2021a)

Crusher Type	Typical Process Stage	Maximum Input Size (mm)	Maximum Output Size (mm)
Gyratory Crusher	Primary	1,500	200-300
Jaw Crusher	Primary	1,400	200-300
Impact Crusher	Primary / Secondary	1,300	200-300
Cone Crusher	Secondary	450	60-80
Cone Crusher	Tertiary	150	<30
Note: 1 mm = 0.0393 in.

Table 2.1 : Maximum Input and Output Sizes for Common Crusher Types (after Jankovic (2015a) )

Figure 2.8 : Typical layout of a multi-stage crushing and screening system. Source: Manufactor (2013a)

Figure 2.9 : A typical stockpile of large-sized aggregates

Aggregate Type	Size	Usage	Description
Fine Aggregates	0.003 in. (0.075 mm) to 0.187 in. (4.75 mm)	mortar, plaster, concrete, asphalt mixture, pavement filling, etc.	sand, fly-ash, fine crushed particles, etc.
Coarse Aggregates	0.187 in. (4.75 mm) to 2.953 in. (75 mm)	concrete, asphalt mixture, pavement base, railway ballast, etc.	gravel, crushed stone, crushed cement concrete
Large-sized Aggregates	¿ 2.953 in. (75 mm)	armor for streambeds, bridge abutments, pilings, and shoreline structures	riprap, jetty stone, cap stone

Table 2.2 : Size and Usage of Typical Aggregate Types

Figure 2.10 : Typical engineering applications of large-sized aggregates. Sources: Lagasse et al. (2006a) ; Shari (2015a) ; Hiller (2017a) ; MagnumStone (2020a) ; Lovas (2021a)

Figure 2.11 : Sieve analysis for regular-sized aggregates

Figure 2.12 : Proportional caliper device for regular-sized aggregates. Source: ASTM D4791

Gradation	Keystone #1 (lbs.)	Keystone #2 (lbs.)	Keystone #3 (lbs.)
RR3	50 (±5)	10 (±1)	1 (±0.1)
RR4	150 (±15)	40 (±4)	1 (±0.1)
RR5	400 (±40)	90 (±13)	3 (±0.1)
RR6	600 (±60)	170 (±17)	6 (±0.5)
RR7	1000 (±100)	300 (±30)	12 (±1)
Note: 1 lb. = 453.6 grams

Table 2.3 : Keystone Requirements for Different Riprap Size/Weight Categories

Gradation	Grid Size (ft.)	Sample Size (Min. Number of Tested Blocks)
RR3	2’ by 25’	2
RR4	3’ by 25’	2
RR5	4’ by 25’	3
RR6	5’ by 30’	3
RR7	5’ by 35’	3
Note: 1 ft. = 30.48 cm

Table 2.4 : Grid Size Requirements for Sampling Different Riprap Gradation Categories

Figure 2.13 : Converted particle-size distribution of IDOT riprap categories RR3 to RR7

Figure 2.14 : (a) keystone-based method to facilitate visual inspection and (b) Tripod weighting tool for individual aggregates

Figure 2.15 : Wolman count method for large-sized aggregates. Source: Bartelt (2018a)

Figure 2.16 : VDG-40 Videograder. Source: Browne et al. (2001a)

Figure 2.17 : Aggregate Imaging System (AIMS). Source: Gates et al. (2011a)

Figure 2.18 : Enhanced-University of Illinois Aggregate Image Analyzer (E-UIAIA). Sources: Tutumluer et al. (2000a) ; Moaveni et al. (2013a)

Figure 2.19 : 3D laser scanning system of aggregates. Sources: Komba et al. (2013a) ; Anochie-Boateng et al. (2013a)

Figure 2.20 : Translucent Segregation Table (TST) for stereo photography of aggregates. Sources: Hryciw et al. (2014a) ; Zheng & Hryciw (2014a)

Figure 2.21 : X-ray CT images of the cylindrical container and the developed 3D models of aggregates. Source: Jin et al. (2018a)

Figure 2.22 : WipFrag software for rock fragmentation analysis. Source: WipWare (2020a)

Figure 2.23 : Watershed segmentation used for railway ballast analysis. Source: Tutumluer et al. (2017a)

Figure 2.24 : Main categories of machine learning and deep learning problems. Source: Nikolenko (2021a)

Figure 2.25 : Typical architecture of CNNs. Source: Alom et al. (2019a)

Figure 2.26 : (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several attention layers running in parallel. Source: Vaswani et al. (2017a)

Figure 2.27 : The transformer model architecture. Source: Vaswani et al. (2017a)

Figure 3.1 : Geographical information for selected IDOT-approved aggregate producers. Source: IDOT (2019b)

Figure 3.2 : Geographical information for the aggregate producers that participated in this study

Aggregate Producer	Aggregate Size Categories	Mineralogy Description	Field Study
Vulcan Materials Company - Lisbon, Illinois	RR3	Dolomite, yellowish	Material Shipped, June 2018 (I) & August 2021 (III)
Prairie Material – Ocoya, Illinois	RR4, RR5, and RR6	Limestone, bluish gray	Inspection Photos Provided (II)
RiverStone Group, Midway Stone – Hillsdale, Illinois	RR6	Dolomite, yellowish to bluish gray, fossiliferous	July 2018 (II)
RiverStone Group, Allied Stone – Milan, Illinois	RR4, RR5, and RR6	Dolomite, white	July 2018 (I, II) & March 2019 (II, III)
Vulcan Materials Company – Kankakee, Illinois	RR4, RR5, RR6, and RR7	Dolomite, white to yellowish	May 2019 (II) & August 2021 (III)

Table 3.1 : Information of Collaborated Aggregate Producers in Illinois

Aggregate Producer	Source Name	Number of Rocks	Size Range (in.)
Vulcan Materials Company, Lisbon, IL	Source 1	40	[3, 6]
RiverStone Group, Milan, IL	Source 2	40	[5, 16]
RiverStone Group, Milan, IL	Source 2–Large	5	[16, 26]
Note: 1 in. = 2.54 cm

Table 3.2 : Material Size and Source Information for the Individual-Aggregate Study

Figure 3.3 : (a) A conceptual illustration, and (b) the actual constructed setup of the field-imaging system for individual aggregates

Figure 3.4 : (a) Manual measurements of riprap dimensions and (b) tripod scale system used by IDOT for measuring the weight of large riprap rocks

Aggregate Producer	Aggregate Size Categories	Number of Images	Rock Description
Prairie Material – Ocoya, Illinois	RR4, RR5, and RR6	6	Limestone, bluish gray, medium-sized*
RiverStone Group, Allied Stone – Milan, Illinois	RR4, RR5, and RR6	14	Dolomite, white, medium-sized*
RiverStone Group, Midway Stone – Hillsdale, Illinois	RR6+	100	Dolomite, yellowish to bluish gray, fossiliferous, large-sized*
Vulcan Materials Company –Kankakee, Illinois	RR4, RR5, RR6, and RR7	44	Dolomite, white to yellowish, large-sized*
* Note: “Medium sized” refers to aggregates weighing between 1 lb. (0.45 kg) and 40 lbs. (18.1 kg);
“Large sized” refers to aggregates weighing between 40 lbs. (18.1 kg) and 600 lbs. (272.2 kg);

Table 3.3 : Source Information and Description of Stockpile Aggregate Image Dataset

Figure 3.5 : (a) Proper positioning of the camera relative to the stockpile, (b) satisfactory image with a calibration ball, (c) satisfactory image without a calibration ball, and (d) unsatisfactory image of a stockpile

Aggregate Producer	Aggregate Size Categories	Mineralogy Description	Number of Collected Samples
Vulcan Materials Company - Lisbon, Illinois	RR3	Dolomite, yellowish	46
RiverStone Group, Allied Stone – Milan, Illinois	RR4	Dolomite, white	20
RiverStone Group, Midway Stone – Hillsdale, Illinois	RR4	Dolomite, white	6
Vulcan Materials Company – Kankakee, Illinois	RR4	Dolomite, white to yellowish	10

Table 3.4 : Source Information of 3D Aggregate Particle Library

Aggregate Source	Aggregate Size Categories	Mineralogy Description	Number of Stockpiles
RR3 Source in 3D Aggregate Particle Library	RR3	Dolomite, yellowish	6
RR4 Sources in 3D Aggregate Particle Library	RR4	Dolomite, white to yellowish	6
Vulcan Materials Company – Kankakee, Illinois	RR4	Dolomite, white to yellowish	3
Vulcan Materials Company – Kankakee, Illinois	RR5	Dolomite, white to yellowish	3

Table 3.5 : Source Information of Aggregate 3D Stockpile Study

Figure 4.1 : Schema of (a) RGB color space, (b) HSV color space, and (c) CIE Lab* color space ( Kothari, 2018a )

Figure 4.2 : Channel images of a large riprap rock studied using CIE Lab* color space

Figure 4.3 : (a) Pixel histogram of color channel and (b) pixel cumulative distribution function of color channel

Figure 4.4 : (a) Original b* channel image and (b) distance map with foreground representative color as the reference

Figure 4.5 : (a) Adaptive thresholding applied image, (b) morphological de-noising applied image, and (c) image-segmentation result

Figure 4.6 : Flowchart of 3D volumetric reconstruction algorithm

Figure 4.7 : Pixel-scope difference between the detected boundary and actual boundary

Figure 4.8 : Effect of rock/ball ratio and calibration ball size on volume overestimation

Figure 4.9 : Comparisons between image analysis results and ground-truth measurements for: (a) Source 1 material with rotate repetitions, (b) Source 1 material after averaging, (c) Source 2 material with rotate repetitions, (d) Source 2 material after averaging, (e) all sources with rotate-repetitions, and (f) all sources after averaging (1 lb. = 0.454 kg, 1

in.^{3}

= 16.4

cm^{3}

)

ID	Measured Weight	Weight Estimated from Hand Measurement	Weight Estimated from Image Analysis	Error Associated with Hand Measurement	Error Associated with Image Analysis
(-)	(lb.)	(lb.)	(lb.)	(%)	(%)
2-1	9.6	10.3	10.0	7.4	4.7
2-2	9.2	17.7	10.5	91.9	14.4
2-3	15.6	50.2	19.0	220.8	21.7
2-4	10.6	18.4	11.6	73.4	9.5
2-5	9.1	16.6	10.0	82.0	9.2

Table 4.1 : Comparisons between Image Analysis Results and Manual Measurements on Source 2 and Source 2-Large Particles (1 lb. = 0.454 kg)

Figure 4.10 : Comparisons between weights estimated from image analyses and weights estimated from hand measurements on Source 2 and Source 2–Large particles (1 lb. = 0.454 kg)

Figure 5.1 : Flowchart of deep learning-based image segmentation and morphological analysis approach

Aggregate Producer	Number of Images	Number of Aggregates
Prairie Material – Ocoya, Illinois	6	520
RiverStone Group, Allied Stone – Milan, Illinois	14	982
RiverStone Group, Midway Stone – Hillsdale, Illinois	100	6,766
Vulcan Materials Company – Kankakee, Illinois	44	3,527
Total	164	11,795

Table 5.1 : Source Information and Description of Stockpile Aggregate Image Dataset

Figure 5.2 : Stockpile aggregate image (a) before labeling and (b) after labeling

Figure 5.3 : Model architecture of the Mask R-CNN framework composed of (a) Region-based Convolutional Neural Network (R-CNN) and (b) Fully Convolutional Network (FCN)

Figure 5.4 : Volume/Weight estimation based on 2D segmented silhouette

Source Name	Number of Particles	Size Range (in.)	Minimum	Maximum	Median	Average
Source 1-RR3	40	3 to 6	1.55	3.26	1.97	2.16
Source 2-RR4	40	5 to 16	1.24	2.94	1.89	1.94
Source 2-RR5	5	16 to 26	1.27	1.9	1.55	1.56
Note: 1 in. = 2.54 cm

Table 5.2 : Flat and Elongated Ratios (3D FER) for Different Riprap Categories in Individual Aggregate Study

Figure 5.5 : Raw images of (a) separated particles, (b) contacting or overlapping particles, and (c) densely stacked particles. Watershed segmented images of (d) separated particles, (e) contacting or overlapping particles, and (f) densely stacked particles. Mask R-CNN segmented images of (g) separated particles, (h) contacting or overlapping particles, and (i) densely stacked particles

Figure 5.6 : (a) Mask R-CNN segmented image (enlarged from Figure 4[i]); (b) histogram distributions and (c) cumulative distribution curves for equivalent particle size; and (d) histogram distributions and (e) cumulative distribution curves for flat and elongated ratio

Figure 5.7 : Completeness and precision metrics used to compare the segmentation results with the ground-truth labeling

ID	Number of Labeled Particles	Number of Segmented Particles	Completeness	Precision
(-)	(-)	(-)	(%)	(%)
1	70	67	95.7	87.7
2	56	53	94.6	88.3
3	131	102	77.9	87.4
4	73	65	89.0	86.1
5	111	92	82.9	83.2

Table 5.3 : Completeness and Precision Results of Randomly Selected Validation Set Images

Figure 6.1 : Equipment setup for 3D reconstruction of aggregates

Figure 6.2 : (a) Multiple view images of an example aggregate particle, (b) salient object masks for each view

Figure 6.3 : Reconstructed sparse point cloud (a) without background suppression and (b) with background suppression

Figure 6.4 : (a) Purple-colored and (b) red-colored object markers for robust point cloud stitching, and (c) background markers for scale reference

Figure 6.5 : (a) Textured model, (b) mesh model, (c) point cloud model of an example aggregate particle, and (d) collage of 40 reconstructed aggregate particles

Rock ID	Surface Area ( $cm^{2}$ )	No. of Vertices	No. of Faces	Resolution ( $points/mm^{2}$ )
1	1308.69	99680	199356	0.76
2	2201.8	209440	418872	0.95
3	2586.81	261948	523884	1.01
4	2108.77	297392	594760	1.41
5	2257.61	151599	303190	0.67
6	1397.78	93369	186734	0.67

Table 6.1 : Surface Resolution and Point Density of Ten Example RR3 Particles

Rock ID	Measured Volume ( $cm^{3}$ )	Reconstructed Volume ( $cm^{3}$ )	Surface Area ( $cm^{2}$ )	Shortest Dimension ( $cm$ )	Intermediate Dimension ( $cm$ )	Longest Dimension ( $cm$ )
1	1014.9	1042.3	685.32	7.682	13.142	22.695
2	763.5	786.33	537.87	9.308	12.519	17.412
3	601.8	605.04	418.69	9.477	10.075	14.572
4	791.4	795.69	558.41	9.118	10.133	19.925
5	727.6	744.83	503.13	9.803	10.649	18.842
6	688.1	691.96	478.72	7.497	9.987	15.925

Table 6.2 : Measured Volume, Reconstructed Volume, Area, and Principal Dimensions of Ten Example RR3 Particles

Figure 6.6 : Comparison of reconstructed volume and measured volume of aggregate samples

Figure 6.7 : (a) Comparison of 3D FER and 2D FER from multiple views, and (b) comparison of 3D sphericity and 2D circularity from multiple views

Figure 6.8 : Comparison of average 2D FER and the ratios of 3D principal dimensions

Figure 7.1 : Optical flow estimation

Figure 7.2 : Ground-truth flow fields and corresponding images in MPI-Sintel optical flow dataset ( Butler et al., 2012a )

Figure 7.3 : (a) FlyingChairs dataset ( Dosovitskiy et al., 2015a ) and (b) FlyingThings3D dataset ( Mayer et al., 2016a )

Figure 7.4 : (a) SceneNet RGB-D synthetic dataset with semantic and instance labels ( McCormac et al., 2017a ) and (b) S3DIS real dataset with semantic and instance labels ( Armeni et al., 2016a )

Simulation Environment	Year	Domain	Engine
Virtual KITTI ( Gaidon et al., 2016a )	2016	Autonomous Driving	Unity
CARLA ( Dosovitskiy et al., 2017a )	2017	Autonomous Driving	Unreal Engine
VRGym ( Xie et al., 2019a )	2019	Robotics	Unreal Engine
ORRB ( Chociej et al., 2019a )	2019	Robotics	Unity

Table 7.1 : A Partial List of Simulation Platforms and Backend Engines for Synthetic Data Generation

Figure 7.5 : Synthetic data generation pipeline for aggregate stockpiles

Figure 7.7 : Fabricated aggregate models in Unity ®

Figure 7.8 : Ground plane and directional light of the stockpile scene

Figure 7.9 : (a) Side view and (b) top view of the multi-view cameras and LiDARs system

Figure 7.10 : (a) Instantiated aggregates before gravity falling and (b) aggregate stockpile after gravity falling

Figure 7.11 : Multi-view camera images of the aggregate stockpile

Figure 7.12 : Cluster of rays cast from one of the LiDARs

Figure 7.13 : Raycasting mechanism

Figure 7.15 : (a) Point cloud and (b) ground-truth instance labels obtained from the raycasting step

Hyper-Parameters		RR3 Stockpile	RR4 Stockpile	RR3-RR4 Mix Stockpile
ROI	$L_{x}\ (m)$	2.0
	$L_{z}\ (m)$	2.0
Aggregates Instantiation	$N_{g}$	9	7	7
	$L_{min}$	6
	$L_{max}$	8
Multi-View Cameras	$N$	36

Table 7.2 : Hyper-Parameters of the Stockpile Scene Control During the Synthetic Data Generation

Scene Type	Number of Scenes	Average Number of Points	Total Number of Instances
RR3 Stockpile	100	202,954	56,486
RR4 Stockpile	100	200,808	23,766
RR3-RR4 Mix Stockpile	100	177,600	24,802
Total	300	-	105,054

Table 7.3 : Statistics of Synthetically Generated Scenes and Point Clouds

Scene Type	Number of Scenes in Train Split	Number of Scenes in Test Split
RR3 Stockpile	90	10
RR4 Stockpile	90	10
RR3-RR4 Mix Stockpile	90	10
Total	270	30

Table 8.1 : Number of Scenes in the Synthetic Dataset Used in Train and Test

Figure 8.1 : PointGroup architecture for instance segmentation

Figure 8.2 : Original coordinates

\mathbb{P}

and shifted coordinates

\mathbb{Q}

by applying the per-point offset

Figure 8.3 : Comparisons of segmentation results and ground-truth instances

Figure 8.4 : Completeness and IoU precision metrics used to compare the segmentation instances with the ground-truth labels

Scene Type	Stockpile ID	Number of Segmented Instances (with IoU ¿ 0.5)	Number of Ground-Truth Instances	Completeness (%)	IoU AP (%)
RR3	1	407	564	72.2	80.5
	2	410	563	72.8	79.4
	3	420	559	75.1	82.1
	4	386	485	79.6	83.6
	5	404	564	71.6	72.5
	6	399	562	71.0	74.0

Table 8.2 : Completeness and IoU AP of the Instance Segmentation Results on Test Set

Figure 9.1 : (a) Coordinate system of the raycasting space and (b) ray endpoints on the disk circumference

Figure 9.2 : (a) Sensor configured with a disk raycasting pattern (b) sensor raycasting on an aggregate model

Figure 9.3 : Sensor sets with increasing number of active sensor views (N)

Figure 9.4 : Collaborative raycasting of two sensors: (a) ray hits on aggregate surface, (b) extracted points of each sensor, and (c) accumulated point clouds from both sensors

Figure 9.5 : Varying-visibility shapes with increasing number of active sensor views (N)

Figure 9.6 : Varying-view shapes at

m^{th}

aggregate model orientation

Dataset Split	Number of Prototype Aggregate Models	Number of Partial-Complete Shape Pairs
Train	82	9,000
Validation	82	184
Test	6	672

Table 9.1 : Dataset Organization for Learning the Shape Completion

Figure 9.7 : SnowflakeNet architecture for 3D shape completion

Figure 9.8 : Effect of the point splitting operation during upsampling

Dataset Split	Chamfer Distance (mm)
Train	0.00391
Validation	0.00483
Test	0.00559

Table 9.2 : Chamfer Distance on Different Dataset Splits

Figure 9.9 : Intermediate and final shape completion results for shapes in the validation set

Figure 9.10 : Comparisons of macro-scale metrics between the completed shapes and ground-truth shapes in the validation set

Figure 9.11 :

Figure 9.12 : Intermediate and final shape completion results for shapes in the test set

Figure 9.13 : Comparisons of macro-scale metrics between the completed shapes and ground-truth shapes in the test set

Figure 9.14 :

Figure 10.1 : Photos of re-engineered RR3 and RR4 stockpiles

Size Category	Number of Aggregate Samples in Stockpile	Number of Stockpiles	Ground-Truth
RR3 (Re-engineered)	46	6	Morphological Properties
RR4 (Re-engineered)	36	6	Morphological Properties
RR3R (Field)	24	3	Weight Measurement
RR4K (Field)	16	3	Weight Measurement
RR5K (Field)	20	3	Weight Measurement

Table 10.1 : Information of Re-engineered and Field Stockpiles

Figure 10.2 : Photos of field RR3R, RR4K and RR5K stockpiles

Figure 10.3 : Field marker system for scale reference

Figure 10.4 : 3D reconstruction results of stockpiles in different categories

Figure 10.5 : 3D instance segmentation results of re-engineered and field stockpiles

Figure 10.6 : Intermediate and final shape completion results for segmented aggregates from the instance segmentation step

Figure 10.7 : Comparisons of morphological properties between the completed aggregates and ground-truth aggregates for all RR3 stockpiles

Figure 10.8 :

Figure 10.9 : Comparisons of morphological properties between the completed aggregates and ground-truth aggregates for all RR4 stockpiles

Figure 10.10 :

Figure 10.11 : Weight comparisons between the completed aggregates and ground-truth measurements for all RR3R, RR4K and RR5K stockpiles

Figure 10.12 : Systematic underestimation in volume for (a) RR3 and RR4 stockpiles and (b) RR3R, RR4K and RR5K stockpiles

Figure 10.13 : Shape Percentage (SP) concept of a partial shape

Figure 10.14 : Shape Percentage (SP, in percentage) analysis of RR3R-S1 stockpile results

Figure 10.15 : Effect of SP thresholding on aggregate dimension and shape metrics (ESD and 3D FER) for re-engineered stockpile data

Figure 10.16 :

Figure 10.17 : Effect of SP thresholding on high-dimensional metrics (surface area and volume) for re-engineered stockpile data

Figure 10.18 :

Figure 10.19 : Effect of SP thresholding on high-dimensional metric (weight) for field stockpile data

Figure 10.20 : Systematic volume/weight underestimation for both re-engineered and field stockpiles (with and without SP thresholding)

Figure 10.21 : Effect of SP thresholding on segmented aggregate shapes in a stockpile (RR4-S6)

Figure 11.1 : Full 3D reconstruction of a larger field stockpile from manually controlled UAV images

Figure 11.2 : (a) Cross-sectional ballast image of a trench cut and (b) segmentation results

Figure 11.3 : 3D reconstruction of the shoulder of a ballasted track

Field imaging framework for morphological characterization of aggregates with computer vision: Algorithms and applications

Abstract

Figures & tables

3D Instance Segmentation

Three-dimensional reconstruction and segmentation of an aggregate stockpile for size and shape analyses

Marker-Based 3D Reconstruction of Aggregates with a Comparative Analysis of 2D and 3D Morphologies

Evaluating Mesh Reconstruction Methods for Crop Phenotyping

Abstract

3D Instance Segmentation

Explore similar work

Three-dimensional reconstruction and segmentation of an aggregate stockpile for size and shape analyses

Marker-Based 3D Reconstruction of Aggregates with a Comparative Analysis of 2D and 3D Morphologies

Evaluating Mesh Reconstruction Methods for Crop Phenotyping