UAMTERS: Uncertainty-Aware Mutation Analysis for DL-enabled Robotic Software

Chengjie Lu; Jiahui Wu; Shaukat Ali; Malaika Din Hashmi; Sebastian Mathias Thomle Mason; Francois Picard; Mikkel Labori Olsen; Thomas Peyrucain

cs.SEFeb 23, 2026

UAMTERS: Uncertainty-Aware Mutation Analysis for DL-enabled Robotic Software

Authors: Chengjie Lu, Jiahui Wu, Shaukat Ali, Malaika Din Hashmi, Sebastian Mathias Thomle Mason, Francois Picard, Mikkel Labori Olsen, Thomas Peyrucain

Organizations: Simula Research Laboratory and University of Oslo · Simula Research Laboratory · Danish Technological Institute · PAL Robotics

Abstract

Self-adaptive robots adjust their behaviors in response to unpredictable environmental changes. These robots often incorporate deep learning (DL) components into their software to support functionality such as perception, decision-making, and control, enhancing autonomy and self-adaptability. However, the inherent uncertainty of DL-enabled software makes it challenging to ensure its dependability in dynamic environments. Consequently, test generation techniques have been developed to test robot software, and classical mutation analysis injects faults into the software to assess the test suite's effectiveness in detecting the resulting failures. However, there is a lack of mutation analysis techniques to assess the effectiveness under the uncertainty inherent to DL-enabled software. To this end, we propose UAMTERS, an uncertainty-aware mutation analysis framework that introduces uncertainty-aware mutation operators to explicitly inject stochastic uncertainty into DL-enabled robotic software, simulating uncertainty in its behavior. We further propose mutation score metrics to quantify a test suite's ability to detect failures under varying levels of uncertainty. We evaluate UAMTERS across three robotic case studies, demonstrating that UAMTERS more effectively distinguishes test suite quality and captures uncertainty-induced failures in DL-enabled software.

Figures & tables

Figure 1: UAMTERS in the Context of the MAPLE-K Loop.

Robot	Model	Model Architecture	Test Suite	Mutation Ratio (MCD)	Mutation Ratio (MCB)
TIAGo Pro	HFDM1	YuNet	lowImg , medImg , highImg	0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50	$0.10\cdot\{1,3,5,7,9\}$ $0.15\cdot\{1,3,5,7,9\}$ $0.20\cdot\{1,3,5,7,9\}$ $0.25\cdot\{1,3,5,7,9\}$ $0.30\cdot\{1,3,5,7,9\}$ $0.35\cdot\{1,3,5,7,9\}$ $0.40\cdot\{1,3,5,7,9\}$ $0.45\cdot\{1,3,5,7,9\}$ $0.50\cdot\{1,3,5,7,9\}$
TIAGo Pro	HFDM2	YuNet-s	lowImg , medImg , highImg
LDR	SCDM	YOLOv11	norImg , expImg
LSRR	STDM1	Faster R-CNN	origImg , daImg , sdImg
	STDM2	Faster R-CNN_v2
	STDM3	RetinaNet

Table 1: Overview of the industrial case studies and experimental configurations, including subject models, test suites, and mutation ratio settings.

RQs	Tasks (Section 4.1 )	Case Study (Section 4.2 )	Metrics (Section 4.4 )	Statistical Tests (Section 4.5 )
RQ1	Compare the proposed MSs with traditional MSs	TIAGo Pro , LDR , LSRR	Img-MS (baseline metrics), Obj-MS , IoU-MS , UA-MS	/
RQ2	Evaluate how effectively the proposed MSs distinguish the quality of different test suites		Obj-MS , IoU-MS , UA-MS	Kruskal–Wallis test, Eta-squared ( $\eta^{2}$ ) effect size
RQ3	Examine how mutant behavior changes under different levels of predictive uncertainty			Spearman’s correlation test Multiple Correlation Coefficient ( $R$ ), F-test

Table 2: Description of tasks, employed case studies, metrics, and statistical tests for each RQ.

Metrics		HFDM 1			HFDM 2			SCDM 1		STDM 1
Metrics		lowImg	medImg	highImg	lowImg	medImg	highImg	norImg	expImg	origImg	daImg	sdImg
Img-MS	$MS_{img}^{miss}$	1.000	1.000	1.000	1.000	1.000	1.000	0.000	1.000	0.481	1.000	1.000
	$MS_{img}^{ghost}$	1.000	1.000	1.000	1.000	1.000	1.000	0.204	0.796	1.000	1.000	1.000
	$MS_{img}^{mg}$	1.000	1.000	1.000	1.000	1.000	1.000	0.204	1.000	1.000	1.000	1.000
Obj-MS	$MS_{obj}^{miss}$	0.499	0.320	0.192	0.482	0.273	0.165	0.000	0.006	0.000	0.015	0.025
	$MS_{obj}^{ghost}$	0.346	0.270	0.175	0.461	0.337	0.217	0.000	0.002	0.009	0.014	0.019

Table 3: Mutation Scores Achieved by Different Test Suites on HFDM 1, HFDM 2, SCDM 1, and STDM 1 – RQ1.

Metrics		STDM 2			STDM 3			STDM 4			STDM 5
Metrics		origImg	daImg	sdImg	origImg	daImg	sdImg	origImg	daImg	sdImg	origImg	daImg	sdImg
Img-MS	$MS_{img}^{miss}$	0.593	1.000	1.000	1.000	1.000	1.000	0.981	0.778	1.000	1.000	1.000	1.000
	$MS_{img}^{ghost}$	0.889	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
	$MS_{img}^{mg}$	0.907	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
Obj-MS	$MS_{obj}^{miss}$	0.002	0.029	0.037	0.010	0.009	0.022	0.005	0.006	0.025	0.274	0.049	0.049
	$MS_{obj}^{ghost}$	0.004	0.012	0.032	0.006	0.012	0.006	0.002	0.005	0.010	0.523	0.474	0.490

Table 4: Mutation Scores Achieved by Different Test Suites on STDM 2, STDM 3, STDM 4, STDM 5 – RQ1.

Metrics		HFDM 1		HFDM 2		SCDM 1		STDM 1		STDM 2		STDM 3		STDM 4		STDM 5
Metrics		$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$	$p$	$\eta^{2}$
Obj-MS	$MS_{obj}^{miss}$	<0.01	0.604	<0.01	0.692	<0.01	0.856	<0.01	0.747	<0.01	0.653	<0.01	0.385	<0.01	0.565	<0.01	0.571
	$MS_{obj}^{ghost}$	<0.01	0.433	<0.01	0.384	<0.01	0.276	<0.01	0.236	<0.01	0.730	<0.01	0.370	<0.01	0.664	0.271	0.004
	$MS_{obj}^{mg}$	<0.01	0.545	<0.01	0.544	<0.01	0.717	<0.01	0.612	<0.01	0.703	<0.01	0.113	<0.01	0.625	<0.01	0.106
IoU-MS	$MS_{iou}$	<0.01	0.639	<0.01	0.712	<0.01	0.132	<0.01	0.185	<0.01	0.310	<0.01	0.054	<0.01	0.212	<0.01	0.294
UA-MS	$MS_{vr}^{match}$	<0.01	0.559	<0.01	0.642	<0.01	0.856	<0.01	0.746	<0.01	0.639	<0.01	0.439	<0.01	0.596	<0.01	0.551

Table 5: Statistical Results Using the Kruskal–Wallis Test and Eta-Squared Effect Size – RQ2. A bold

\eta^{2}

with

p<0.01

indicates a statistically significant difference. The

\eta^{2}

effect sizes are further interpreted as: Large if

\eta^{2}\geq 0.14

, Moderate if

0.06\leq\eta^{2}<0.14

, and Small if

0.01\leq\eta^{2}<0.06

Figure 3: Mutation Scores Achieved by Test Suites with 95% Confidence Intervals for Different Dropout Rates (Mutation Operator: MCD , Mutation Scores: Obj-MS , IoU-MS , and UA-MS derived from

\mathcal{S}_{match}

) – RQ3.

Figure 4: Mutation Scores Achieved by Test Suites with 95% Confidence Intervals for Different Dropout Rates (Mutation Operator: MCD , Mutation Scores: UA-MS Derived from

\mathcal{S}_{miss}

and

\mathcal{S}_{ghost}

) – RQ3.

Metrics		HFDM 1		HFDM 2		SCDM 1		STDM 1		STDM 2		STDM 3		STDM 4		STDM 5
Metrics		$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$	$p$	$\rho$
Obj-MS	$MS_{obj}^{miss}$	<0.01	1.000	<0.01	1.000	<0.01	0.917	<0.01	0.933	<0.01	0.983	<0.01	0.800	<0.01	0.867	<0.01	1.000
	$MS_{obj}^{ghost}$	<0.01	1.000	<0.01	1.000	<0.01	0.877	<0.01	1.000	<0.01	0.917	<0.01	0.983	<0.01	0.933	<0.01	1.000
	$MS_{obj}^{mg}$	<0.01	1.000	<0.01	1.000	<0.01	0.883	<0.01	0.983	<0.01	1.000	<0.01	0.950	<0.01	0.950	<0.01	1.000
IoU-MS	$MS_{iou}$	<0.01	1.000	<0.01	1.000	<0.01	1.000	<0.01	1.000	<0.01	1.000	<0.01	1.000	<0.01	1.000	<0.01	1.000
UA-MS	$MS_{vr}^{match}$	<0.01	1.000	<0.01	1.000	<0.01	0.917	<0.01	0.983	<0.01	0.983	<0.01	0.800	<0.01	0.933	<0.01	0.917

Table 6: Correlation Test Results between Dropout Rates and Mutation Scores for MC-Dropout-based Mutation Operator Using Spearman’s Rank Correlation (

\rho

) test – RQ3. Bold values indicate statistically significant correlations (

p<0.01

). The

\rho

values are further interpreted as: Very High if

\rho

\in

[0.900, 1.000] or [-1.000, -0.900], High if

\rho

\in

[0.700, 0.900) or (-0.900, -0.700], Moderate if

\rho

\in

[0.50, 0.700) or (-0.700, 0.500], and Low if

\rho

\in

[0.300, 0.500) or (-0.500, -0.300].

Figure 5: Mutation Scores Achieved by Test Suites with 95% Confidence Intervals for Dropout Rates

\times

Block Sizes (Mutation Operator: MCB , Mutation Scores: Obj-MS , IoU-MS , and UA-MS derived from

\mathcal{S}_{match}

) – RQ3.

Figure 6: Mutation Scores Achieved by Test Suites with 95% Confidence Intervals for Dropout Rates

\times

Block Sizes (Mutation Operator: MCB , Mutation Scores: UA-MS Derived from

\mathcal{S}_{miss}

and

\mathcal{S}_{ghost}

) – RQ3.

Metrics		HFDM 1		HFDM 2		SCDM 1		STDM 1		STDM 2		STDM 3		STDM 4		STDM 5
Metrics		$p$	$R$	$p$	$R$	$p$	$R$	$p$	$R$	$p$	$R$	$p$	$R$	$p$	$R$	$p$	$R$
Obj-MS	$MS_{obj}^{miss}$	<0.01	0.974	<0.01	0.976	<0.01	0.814	<0.01	0.951	<0.01	0.805	<0.01	0.751	<0.01	0.829	<0.01	0.984
	$MS_{obj}^{ghost}$	<0.01	0.976	<0.01	0.985	<0.01	0.856	<0.01	0.952	<0.01	0.966	<0.01	0.898	0.013	0.432	<0.01	0.917
	$MS_{obj}^{mg}$	<0.01	0.969	<0.01	0.979	<0.01	0.846	<0.01	0.966	<0.01	0.903	<0.01	0.800	<0.01	0.849	<0.01	0.924
IoU-MS	$MS_{iou}$	<0.01	0.957	<0.01	0.968	<0.01	0.987	<0.01	0.977	<0.01	0.915	<0.01	0.898	<0.01	0.951	<0.01	0.977
UA-MS	$MS_{vr}^{match}$	<0.01	0.970	<0.01	0.975	<0.01	0.792	<0.01	0.930	<0.01	0.764	<0.01	0.736	<0.01	0.834	<0.01	0.947

Table 7: Multiple Correlation Test Results between (Dropout Rate, Block Size) and Mutation Scores for MCB Operator Using Multiple Correlation Coefficient (

R

) – RQ3. Bold values indicate statistically significant correlations (

p<0.01

). The

R

values are further interpreted as: Very High if

R

\in

[0.900, 1.000], High if

R

\in

[0.700, 0.900), Moderate if

R

\in

[0.50, 0.700), and Low if

R

\in

[0.300, 0.500).

Mutation Testing

Mutation testing evaluates test suites by injecting controlled faults and measuring whether tests detect them.

See all 18 papers

May 4, 2026

Human-in-the-Loop Uncertainty Analysis in Self-Adaptive Robots Using LLMs

Self-adaptive robots operate in dynamic, unpredictable environments where unaddressed uncertainties can lead to safety violations and operational failures. However, systematically identifying and analyzing these uncertainties, including their sources, impacts, and mitigation strategies, remains a significant challenge given the inherent complexity of real-world environments, dynamic robotic behavior, and the rapid evolution of robotic technologies. To address this, we introduce RoboULM, a human-in-the-loop methodology and tool that supports practitioners in systematically exploring uncertainties at the design stage using large language models (LLMs). Moreover, we present an uncertainty taxonomy that provides a detailed catalog of uncertainties in self-adaptive robots. We evaluated RoboULM with 16 practitioners from four industrial use cases. The results show that RoboULM was perceived as both useful and easy to understand, with the participants particularly valuing structured prompting and iterative refinement support. These findings demonstrate the potential of RoboULM as a viable solution for systematic uncertainty analysis in complex robots.

Hassan Sartaj, Jalil Boudjadar, Mirgita Frasheri +2

Simula Research Laboratory · Aarhus University

Robotics

Ask PDF

Jul 2, 2026

UA-ChatDev: Uncertainty-Aware Multi-Agent Collaboration for Reliable Software Development

Software development is a complex task that demands cooperation among agents with diverse roles. Large language models (LLMs) have enabled autonomous multi-agent software development frameworks that leverage role-based collaboration to automate requirements analysis, coding, testing, and refinement. However, existing approaches typically assume that intermediate agent outputs are equally reliable, leaving them vulnerable to hallucination propagation, where incorrect decisions generated in early development phases are transferred to downstream agents and negatively impact final software quality. To address this challenge, we propose UA-ChatDev, an uncertainty-aware multi-agent software development framework that integrates uncertainty quantification into agent interactions. It introduces a lightweight uncertainty estimation mechanism based on token-level log probabilities to assess the confidence of agent responses and employs phase-aware threshold calibration to selectively trigger retrieval-based verification when uncertainty exceeds acceptable levels. Extensive experiments on the SRDD benchmark demonstrate that UA-ChatDev consistently outperforms existing single-agent and multi-agent software development frameworks across completeness, executability, consistency, and overall quality metrics. Further ablation studies and communication analyses verify that uncertainty-aware interactions enhance code execution reliability.

Temitayo Olamilekan Ogunsusi, Lijun Qian, Xishuang Dong

Department of Electrical and Computer Engineering Prairie View A&M University, Prairie View, TX 77446, USA

Software Engineering Agents Multi-Agent Collaboration Multi-Agent Coding LLM Agent Reliability

Ask PDF

Mar 5, 2026

SPIRIT: Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty

Deep learning (DL) has enabled impressive advances in robotic perception, yet its limited robustness and lack of interpretability hinder reliable deployment in safety critical applications. We propose a concept termed perceptive shared autonomy, in which uncertainty estimates from DL based perception are used to regulate the level of autonomy. Specifically, when the robot's perception is confident, semi-autonomous manipulation is enabled to improve performance; when uncertainty increases, control transitions to haptic teleoperation for maintaining robustness. In this way, high-performing but uninterpretable DL methods can be integrated safely into robotic systems. A key technical enabler is an uncertainty aware DL based point cloud registration approach based on the so called Neural Tangent Kernels (NTK). We evaluate perceptive shared autonomy on challenging aerial manipulation tasks through a user study of 15 participants and realization of mock-up industrial scenarios, demonstrating reliable robotic manipulation despite failures in DL based perception. The resulting system, named SPIRIT, improves both manipulation performance and system reliability. SPIRIT was selected as a finalist of a major industrial innovation award.

Jongseok Lee, Ribin Balachandran, Harsimran Singh +6

DLR · KIT · TU Munich

Robotic Manipulation Robotic Perception Uncertainty Quantification

Ask PDF

Abstract

Mutation Testing

Explore similar work

Human-in-the-Loop Uncertainty Analysis in Self-Adaptive Robots Using LLMs

UA-ChatDev: Uncertainty-Aware Multi-Agent Collaboration for Reliable Software Development

SPIRIT: Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty