Digital Twin Driven Textile Classification and Foreign Object Recognition in Automated Sorting Systems

Serkan Ergun; Tobias Mitterer; Hubert Zangl

cs.CVMar 5, 2026

Digital Twin Driven Textile Classification and Foreign Object Recognition in Automated Sorting Systems

Authors: Serkan Ergun, Tobias Mitterer, Hubert Zangl

Organizations: Institute of Smart Systems Technologies, University of Klagenfurt, Klagenfurt am Wörthersee, A 9020 · AAU SAL USE Laboratory, Silicon Austria Labs, Klagenfurt am Wörthersee, A 9020

Abstract

The increasing demand for sustainable textile recycling requires robust automation solutions capable of handling deformable garments and detecting foreign objects in cluttered environments. This work presents a digital twin driven robotic sorting system that integrates grasp prediction, multi modal perception, and semantic reasoning for real world textile classification. A dual arm robotic cell equipped with RGBD sensing, capacitive tactile feedback, and collision-aware motion planning autonomously separates garments from an unsorted basket, transfers them to an inspection zone, and classifies them using state of the art Visual Language Models (VLMs). We benchmark nine VLM s from five model families on a dataset of 223 inspection scenarios comprising shirts, socks, trousers, underwear, foreign objects (including garments outside of the aforementioned classes), and empty scenes. The evaluation assesses per class accuracy, hallucination behavior, and computational performance under practical hardware constraints. Results show that the Qwen model family achieves the highest overall accuracy (up to 87.9 %), with strong foreign object detection performance, while lighter models such as Gemma3 offer competitive speed accuracy trade offs for edge deployment. A digital twin combined with MoveIt enables collision aware path planning and integrates segmented 3D point clouds of inspected garments into the virtual environment for improved manipulation reliability. The presented system demonstrates the feasibility of combining semantic VLM reasoning with conventional grasp detection and digital twin technology for scalable, autonomous textile sorting in realistic industrial settings.

Figures & tables

Figure 1: An overview of the experimental setup and the corresponding digitial twin in RViz.

Figure 2: Schematic of our framework, consisting of two UR7e robots ( Alice and Bob with Robotiq 2F-140 grippers, Alice uses CapTac as fingertips.), two home grade desktop PC’s equipped with Nvidia RTX 3060 budget consumer graphics cards, one Nvidia H200X professional graphics card and two Intel Realsense cameras for grasp detection and object classification.

Figure 3: Flowchart of the textile inspection process.

Panel of Figure 4 — Figure 4: Two exemplary images of garments in zone B . To consider robustness in cluttered environments, additional smaller garments were placed on the ground and several distracting items were present. The response of each model is given together with its ground truth.

	Overall	Shirt	Sock	Trousers	Underwear	Other	Empty
Image Count	223	38	64	43	12	65	4
Models	Accuracy
gemma3:12b	$76.23\text{\,}\mathrm{\%}$	$55.26\text{\,}\mathrm{\%}$	$95.31\text{\,}\mathrm{\%}$	$67.44\text{\,}\mathrm{\%}$	$50.00\text{\,}\mathrm{\%}$	$76.92\text{\,}\mathrm{\%}$	$100.00\text{\,}\mathrm{\%}$
llama3.2-vision:90b	$60.09\text{\,}\mathrm{\%}$	$18.42\text{\,}\mathrm{\%}$	$87.50\text{\,}\mathrm{\%}$	$34.88\text{\,}\mathrm{\%}$	$58.33\text{\,}\mathrm{\%}$	$73.85\text{\,}\mathrm{\%}$	$25.00\text{\,}\mathrm{\%}$
llama4:16x17b	$71.30\text{\,}\mathrm{\%}$	$31.58\text{\,}\mathrm{\%}$	$89.06\text{\,}\mathrm{\%}$	$60.47\text{\,}\mathrm{\%}$	$50.00\text{\,}\mathrm{\%}$	$89.23\text{\,}\mathrm{\%}$	$00.00\text{\,}\mathrm{\%}$
llava:34b	$50.67\text{\,}\mathrm{\%}$	$23.68\text{\,}\mathrm{\%}$	$76.56\text{\,}\mathrm{\%}$	$2.33\text{\,}\mathrm{\%}$	$41.67\text{\,}\mathrm{\%}$	$72.31\text{\,}\mathrm{\%}$	$50.00\text{\,}\mathrm{\%}$

Table 1: Performance Benchmark for all investigated models.

Panel of Figure 5 — Figure 5: Confusion matrices for all nine considered VLM .

	gemma3:12b	llama3.2-vision:90b	llama4:16x17b	qwen3-vl:235b	llava:34b
GPU	H200
$\overline{t}$	0.653	0.953	0.969	2.444	0.409
$P_{10}$	0.620	0.620	0.904	1.739	0.378
$P_{90}$	0.689	0.690	1.016	3.072	0.411

Table 2: Computation times in

\text{\,}\mathrm{s}

for all investigated models: average

\overline{t}

10^{th}

percentile

P_{10}

and

90^{th}

percentile

P_{90}

Classification

Classification maps observations to discrete labels, including familiar, unknown, hierarchical, or multiple categories.

See all 408 papers

Mar 4, 2026

GarmentPile++: Affordance-Driven Cluttered Garments Retrieval with Vision-Language Reasoning

Garment manipulation has attracted increasing attention due to its critical role in home-assistant robotics. However, the majority of existing garment manipulation works assume an initial state consisting of only one garment, while piled garments are far more common in real-world settings. To bridge this gap, we propose a novel garment retrieval pipeline that can not only follow language instruction to execute safe and clean retrieval but also guarantee exactly one garment is retrieved per attempt, establishing a robust foundation for the execution of downstream tasks (e.g., folding, hanging, wearing). Our pipeline seamlessly integrates vision-language reasoning with visual affordance perception, fully leveraging the high-level reasoning and planning capabilities of VLMs alongside the generalization power of visual affordance for low-level actions. To enhance the VLM's comprehensive awareness of each garment's state within a garment pile, we employ visual segmentation model (SAM2) to execute object segmentation on the garment pile for aiding VLM-based reasoning with sufficient visual cues. A mask fine-tuning mechanism is further integrated to address scenarios where the initial segmentation results are suboptimal. In addition, a dual-arm cooperation framework is deployed to address cases involving large or long garments, as well as excessive garment sagging caused by incorrect grasping point determination, both of which are strenuous for a single arm to handle. The effectiveness of our pipeline are consistently demonstrated across diverse tasks and varying scenarios in both real-world and simulation environments. Project page: https://garmentpile2.github.io/.

Mingleyang Li, Yuran Wang, Yue Chen +6

Peking University · The University of Hong Kong · Northwestern University

Affordance-Based Robotic Manipulation Robotic Manipulation Deformable Object Manipulation

Ask PDF

Mar 11, 2026

Learning Bimanual Cloth Manipulation with Vision-based Tactile Sensing via Single Robotic Arm

Robotic cloth manipulation remains challenging due to the high-dimensional state space of fabrics, their deformable nature, and frequent occlusions that limit vision-based sensing. Although dual-arm systems can mitigate some of these issues, they increase hardware and control complexity. This paper presents Touch G.O.G., a compact vision-based tactile gripper and perception/control framework for single-arm bimanual cloth manipulation. The proposed framework combines three key components: (1) a novel gripper design and control strategy for in-gripper cloth sliding with a single robot arm, (2) a Vision Foundation Model-backboned Vision Transformer pipeline for cloth part classification (PC-Net) and edge pose estimation (PE-Net) using real and synthetic tactile images, and (3) an encoder-decoder synthetic data generator (SD-Net) that reduces manual annotation by producing high-fidelity tactile images. Experiments show 96% accuracy in distinguishing edges, corners, interior regions, and grasp failures, together with sub-millimeter edge localization and 4.5° orientation error. Real-world results demonstrate reliable cloth unfolding, even for crumpled fabrics, using only a single robotic arm. These results highlight Touch G.O.G. as a compact and cost-effective solution for deformable object manipulation.

Dongmyoung Lee, Wei Chen, Xiaoshuai Chen +2

Robot Intelligence Lab, Dyson School of Design Engineering, Imperial College London, 25 Exhibition Road, London, SW7 2DB, UK

Deformable Object Manipulation Bimanual Robotic Manipulation Vision-Based Tactile Sensing Robotic Hand Design

Ask PDF

Aug 28, 2026

Training-free Suction Grasp Detection for Deformed Aseptic Cartons Using Vision-Language Models and Geometric Surface Scoring

Robotic sorting of recyclable waste is challenging due to the deformable and geometrically inconsistent nature of target objects. We present a training-free suction grasping system for sorting deformed aseptic beverage cartons, decoupling target identification from grasp-point selection. An open-vocabulary vision-language model detects cartons from a text prompt, SAM2 refines each detection into an instance mask, and a geometric scoring method selects the suction point by combining surface flatness with normal alignment. Three geometric methods are compared: k-nearest-neighbour PCA, Sobel cross-product, and RANSAC plane fitting. Evaluated on a real robot across three deformation levels and 35 cluttered scenes, single-object grasp success reaches 88.2% and end-to-end retrieval in clutter is 72.6%.

Marin Maletic, Goran Vasiljevic

Laboratory for Robotics and Intelligent Control Systems University of Zagreb Faculty of Electrical Engineering and Computing, Unska 3, 10000 Zagreb, Croatia

Robotic Grasping Robotics Deformable Object Manipulation

Ask PDF

Abstract

Classification

Explore similar work

GarmentPile++: Affordance-Driven Cluttered Garments Retrieval with Vision-Language Reasoning

Learning Bimanual Cloth Manipulation with Vision-based Tactile Sensing via Single Robotic Arm

Training-free Suction Grasp Detection for Deformed Aseptic Cartons Using Vision-Language Models and Geometric Surface Scoring