UAMTERS: Uncertainty-Aware Mutation Analysis for DL-enabled Robotic Software

Chengjie Lu; Jiahui Wu; Shaukat Ali; Malaika Din Hashmi; Sebastian Mathias Thomle Mason; Francois Picard; Mikkel Labori Olsen; Thomas Peyrucain

cs.SEFeb 23, 2026

UAMTERS: Uncertainty-Aware Mutation Analysis for DL-enabled Robotic Software

Authors: Chengjie Lu, Jiahui Wu, Shaukat Ali, Malaika Din Hashmi, Sebastian Mathias Thomle Mason, Francois Picard, Mikkel Labori Olsen, Thomas Peyrucain

Organizations: Simula Research Laboratory and University of Oslo · Simula Research Laboratory · Danish Technological Institute · PAL Robotics

Abstract

Self-adaptive robots adjust their behaviors in response to unpredictable environmental changes. These robots often incorporate deep learning (DL) components into their software to support functionality such as perception, decision-making, and control, enhancing autonomy and self-adaptability. However, the inherent uncertainty of DL-enabled software makes it challenging to ensure its dependability in dynamic environments. Consequently, test generation techniques have been developed to test robot software, and classical mutation analysis injects faults into the software to assess the test suite's effectiveness in detecting the resulting failures. However, there is a lack of mutation analysis techniques to assess the effectiveness under the uncertainty inherent to DL-enabled software. To this end, we propose UAMTERS, an uncertainty-aware mutation analysis framework that introduces uncertainty-aware mutation operators to explicitly inject stochastic uncertainty into DL-enabled robotic software, simulating uncertainty in its behavior. We further propose mutation score metrics to quantify a test suite's ability to detect failures under varying levels of uncertainty. We evaluate UAMTERS across three robotic case studies, demonstrating that UAMTERS more effectively distinguishes test suite quality and captures uncertainty-induced failures in DL-enabled software.

Figures & tables

Figure 1: UAMTERS in the Context of the MAPLE-K Loop.

Robot	Model	Model Architecture	Test Suite	Mutation Ratio (MCD)	Mutation Ratio (MCB)
TIAGo Pro	HFDM1	YuNet	lowImg , medImg , highImg	0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50	$0.10\cdot\{1,3,5,7,9\}$ $0.15\cdot\{1,3,5,7,9\}$ $0.20\cdot\{1,3,5,7,9\}$ $0.25\cdot\{1,3,5,7,9\}$ $0.30\cdot\{1,3,5,7,9\}$ $0.35\cdot\{1,3,5,7,9\}$ $0.40\cdot\{1,3,5,7,9\}$ $0.45\cdot\{1,3,5,7,9\}$ $0.50\cdot\{1,3,5,7,9\}$
TIAGo Pro	HFDM2	YuNet-s	lowImg , medImg , highImg
LDR	SCDM	YOLOv11	norImg , expImg
LSRR	STDM1	Faster R-CNN	origImg , daImg , sdImg
	STDM2	Faster R-CNN_v2
	STDM3	RetinaNet

Table 1: Overview of the industrial case studies and experimental configurations, including subject models, test suites, and mutation ratio settings.

RQs	Tasks (Section 4.1 )	Case Study (Section 4.2 )	Metrics (Section 4.4 )	Statistical Tests (Section 4.5 )
RQ1	Compare the proposed MSs with traditional MSs	TIAGo Pro , LDR , LSRR	Img-MS (baseline metrics), Obj-MS , IoU-MS , UA-MS	/
RQ2	Evaluate how effectively the proposed MSs distinguish the quality of different test suites		Obj-MS , IoU-MS , UA-MS	Kruskal–Wallis test, Eta-squared ( $\eta^{2}$ ) effect size
RQ3	Examine how mutant behavior changes under different levels of predictive uncertainty			Spearman’s correlation test Multiple Correlation Coefficient ( $R$ ), F-test

Table 2: Description of tasks, employed case studies, metrics, and statistical tests for each RQ.

Metrics		HFDM 1			HFDM 2			SCDM 1		STDM 1
Metrics		lowImg	medImg	highImg	lowImg	medImg	highImg	norImg	expImg	origImg	daImg	sdImg
Img-MS	$MS_{img}^{miss}$	1.000	1.000	1.000	1.000	1.000	1.000	0.000	1.000	0.481	1.000	1.000
	$MS_{img}^{ghost}$	1.000	1.000	1.000	1.000	1.000	1.000	0.204	0.796	1.000	1.000	1.000
	$MS_{img}^{mg}$	1.000	1.000	1.000	1.000	1.000	1.000	0.204	1.000	1.000	1.000	1.000
Obj-MS	$MS_{obj}^{miss}$	0.499	0.320	0.192	0.482	0.273	0.165	0.000	0.006	0.000	0.015	0.025
	$MS_{obj}^{ghost}$	0.346	0.270	0.175	0.461	0.337	0.217	0.000	0.002	0.009	0.014	0.019

Table 3: Mutation Scores Achieved by Different Test Suites on HFDM 1, HFDM 2, SCDM 1, and STDM 1 – RQ1.

Metrics		STDM 2			STDM 3			STDM 4			STDM 5
Metrics		origImg	daImg	sdImg	origImg	daImg	sdImg	origImg	daImg	sdImg	origImg	daImg	sdImg
Img-MS	$MS_{img}^{miss}$	0.593	1.000	1.000	1.000	1.000	1.000	0.981	0.778	1.000	1.000	1.000	1.000
	$MS_{img}^{ghost}$	0.889	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
	$MS_{img}^{mg}$	0.907	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
Obj-MS	$MS_{obj}^{miss}$	0.002	0.029	0.037	0.010	0.009	0.022	0.005	0.006	0.025	0.274	0.049	0.049
	$MS_{obj}^{ghost}$	0.004	0.012	0.032	0.006	0.012	0.006	0.002	0.005	0.010	0.523	0.474	0.490

Table 4: Mutation Scores Achieved by Different Test Suites on STDM 2, STDM 3, STDM 4, STDM 5 – RQ1.

Metrics		HFDM 1		HFDM 2		SCDM 1		STDM 1		STDM 2		STDM 3		STDM 4		STDM 5
Metrics		$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$
Obj-MS	$MS_{obj}^{miss}$	<0.01	0.604	<0.01	0.692	<0.01	0.856	<0.01	0.747	<0.01	0.653	<0.01	0.385	<0.01	0.565	<0.01	0.571
	$MS_{obj}^{ghost}$	<0.01	0.433	<0.01	0.384	<0.01	0.276	<0.01	0.236	<0.01	0.730	<0.01	0.370	<0.01	0.664	0.271	0.004
	$MS_{obj}^{mg}$	<0.01	0.545	<0.01	0.544	<0.01	0.717	<0.01	0.612	<0.01	0.703	<0.01	0.113	<0.01	0.625	<0.01	0.106
IoU-MS	$MS_{iou}$	<0.01	0.639	<0.01	0.712	<0.01	0.132	<0.01	0.185	<0.01	0.310	<0.01	0.054	<0.01	0.212	<0.01	0.294
UA-MS	$MS_{vr}^{match}$	<0.01	0.559	<0.01	0.642	<0.01	0.856	<0.01	0.746	<0.01	0.639	<0.01	0.439	<0.01	0.596	<0.01	0.551

Table 5: Statistical Results Using the Kruskal–Wallis Test and Eta-Squared Effect Size – RQ2. A bold

\eta^{2}

with

p<0.01

indicates a statistically significant difference. The

\eta^{2}

effect sizes are further interpreted as: Large if

\eta^{2}\geq 0.14

, Moderate if

0.06\leq\eta^{2}<0.14

, and Small if

0.01\leq\eta^{2}<0.06

Figure 3: Mutation Scores Achieved by Test Suites with 95% Confidence Intervals for Different Dropout Rates (Mutation Operator: MCD , Mutation Scores: Obj-MS , IoU-MS , and UA-MS derived from

\mathcal{S}_{match}

) – RQ3.

Figure 4: Mutation Scores Achieved by Test Suites with 95% Confidence Intervals for Different Dropout Rates (Mutation Operator: MCD , Mutation Scores: UA-MS Derived from

\mathcal{S}_{miss}

and

\mathcal{S}_{ghost}

) – RQ3.

Metrics		HFDM 1		HFDM 2		SCDM 1		STDM 1		STDM 2		STDM 3		STDM 4		STDM 5
Metrics		$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$
Obj-MS	$MS_{obj}^{miss}$	<0.01	1.000	<0.01	1.000	<0.01	0.917	<0.01	0.933	<0.01	0.983	<0.01	0.800	<0.01	0.867	<0.01	1.000
	$MS_{obj}^{ghost}$	<0.01	1.000	<0.01	1.000	<0.01	0.877	<0.01	1.000	<0.01	0.917	<0.01	0.983	<0.01	0.933	<0.01	1.000
	$MS_{obj}^{mg}$	<0.01	1.000	<0.01	1.000	<0.01	0.883	<0.01	0.983	<0.01	1.000	<0.01	0.950	<0.01	0.950	<0.01	1.000
IoU-MS	$MS_{iou}$	<0.01	1.000	<0.01	1.000	<0.01	1.000	<0.01	1.000	<0.01	1.000	<0.01	1.000	<0.01	1.000	<0.01	1.000
UA-MS	$MS_{vr}^{match}$	<0.01	1.000	<0.01	1.000	<0.01	0.917	<0.01	0.983	<0.01	0.983	<0.01	0.800	<0.01	0.933	<0.01	0.917

Table 6: Correlation Test Results between Dropout Rates and Mutation Scores for MC-Dropout-based Mutation Operator Using Spearman’s Rank Correlation (

\rho

) test – RQ3. Bold values indicate statistically significant correlations (

p<0.01

). The

\rho

values are further interpreted as: Very High if

\rho

\in

[0.900, 1.000] or [-1.000, -0.900], High if

\rho

\in

[0.700, 0.900) or (-0.900, -0.700], Moderate if

\rho

\in

[0.50, 0.700) or (-0.700, 0.500], and Low if

\rho

\in

[0.300, 0.500) or (-0.500, -0.300].

Figure 5: Mutation Scores Achieved by Test Suites with 95% Confidence Intervals for Dropout Rates

\times

Block Sizes (Mutation Operator: MCB , Mutation Scores: Obj-MS , IoU-MS , and UA-MS derived from

\mathcal{S}_{match}

) – RQ3.

Figure 6: Mutation Scores Achieved by Test Suites with 95% Confidence Intervals for Dropout Rates

\times

Block Sizes (Mutation Operator: MCB , Mutation Scores: UA-MS Derived from

\mathcal{S}_{miss}

and

\mathcal{S}_{ghost}

) – RQ3.

Metrics		HFDM 1		HFDM 2		SCDM 1		STDM 1		STDM 2		STDM 3		STDM 4		STDM 5
Metrics		$p$	$R$	$p$	$R$	$p$	$R$	$p$	$R$	$p$	$R$	$p$	$R$	$p$	$R$	$p$	$R$
Obj-MS	$MS_{obj}^{miss}$	<0.01	0.974	<0.01	0.976	<0.01	0.814	<0.01	0.951	<0.01	0.805	<0.01	0.751	<0.01	0.829	<0.01	0.984
	$MS_{obj}^{ghost}$	<0.01	0.976	<0.01	0.985	<0.01	0.856	<0.01	0.952	<0.01	0.966	<0.01	0.898	0.013	0.432	<0.01	0.917
	$MS_{obj}^{mg}$	<0.01	0.969	<0.01	0.979	<0.01	0.846	<0.01	0.966	<0.01	0.903	<0.01	0.800	<0.01	0.849	<0.01	0.924
IoU-MS	$MS_{iou}$	<0.01	0.957	<0.01	0.968	<0.01	0.987	<0.01	0.977	<0.01	0.915	<0.01	0.898	<0.01	0.951	<0.01	0.977
UA-MS	$MS_{vr}^{match}$	<0.01	0.970	<0.01	0.975	<0.01	0.792	<0.01	0.930	<0.01	0.764	<0.01	0.736	<0.01	0.834	<0.01	0.947

Table 7: Multiple Correlation Test Results between (Dropout Rate, Block Size) and Mutation Scores for MCB Operator Using Multiple Correlation Coefficient (

R

) – RQ3. Bold values indicate statistically significant correlations (

p<0.01

). The

R

values are further interpreted as: Very High if

R

\in

[0.900, 1.000], High if

R

\in

[0.700, 0.900), Moderate if

R

\in

[0.50, 0.700), and Low if

R

\in

[0.300, 0.500).

Mutation Testing

Mutation testing evaluates test suites by injecting controlled faults and measuring whether tests detect them.

See all 18 papers

Human-in-the-Loop Uncertainty Analysis in Self-Adaptive Robots Using LLMs
May 4, 2026Hassan Sartaj, Jalil Boudjadar, Mirgita Frasheri +2Robotics
Simula Research Laboratory · Aarhus University
UA-ChatDev: Uncertainty-Aware Multi-Agent Collaboration for Reliable Software Development
Jul 2, 2026Temitayo Olamilekan Ogunsusi, Lijun Qian, Xishuang DongSoftware Engineering Agents Multi-Agent Collaboration
Department of Electrical and Computer Engineering Prairie View A&M University, Prairie View, TX 77446, USA
SPIRIT: Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty
Mar 5, 2026Jongseok Lee, Ribin Balachandran, Harsimran Singh +6Robotic Manipulation Robotic Perception
DLR · KIT · TU Munich

May 4, 2026J/K move · Enter open · S save

Human-in-the-Loop Uncertainty Analysis in Self-Adaptive Robots Using LLMs

Hassan Sartaj, Jalil Boudjadar, Mirgita Frasheri, Shaukat Ali+1

Simula Research Laboratory · Aarhus University

Robotics

Open paper AskPDF arXiv

Self-adaptive robots operate in dynamic, unpredictable environments where unaddressed uncertainties can lead to safety violations and operational failures. However, systematically identifying and analyzing these uncertainties, including their sources, impacts, and mitigation strategies, remains a significant challenge given the inherent complexity of real-world environments, dynamic robotic behavior, and the rapid evolution of robotic technologies. To address this, we introduce RoboULM, a human-in-the-loop methodology and tool that supports practitioners in systematically exploring uncertainties at the design stage using large language models (LLMs). Moreover, we present an uncertainty taxonomy that provides a detailed catalog of uncertainties in self-adaptive robots. We evaluated RoboULM with 16 practitioners from four industrial use cases. The results show that RoboULM was perceived as both useful and easy to understand, with the participants particularly valuing structured prompting and iterative refinement support. These findings demonstrate the potential of RoboULM as a viable solution for systematic uncertainty analysis in complex robots.

Abstract

Mutation Testing

Explore similar work

Human-in-the-Loop Uncertainty Analysis in Self-Adaptive Robots Using LLMs

UA-ChatDev: Uncertainty-Aware Multi-Agent Collaboration for Reliable Software Development

SPIRIT: Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty