Adapting Vision-Language Models for Human-Readable XAI in Industrial Object Detection

Sarvenaz Sardari; Freddy Fernandes; Samarth Yelvande; Jose Moises Araya-Martinez; Alina Roitberg

cs.CVSep 17, 2026

Adapting Vision-Language Models for Human-Readable XAI in Industrial Object Detection

Authors: Sarvenaz Sardari, Freddy Fernandes, Samarth Yelvande, Jose Moises Araya-Martinez, Alina Roitberg

Organizations: Mercedes-Benz AG, Future Automotive Manufacturing, Benz-Str. Bau40, 71063 Sindelfingen, Germany · University of Stuttgart, AI, Intelligent Sensing and Perception, Universitätsstr. 38, 70569 Stuttgart, Germany

Abstract

Explainable Artificial Intelligence (XAI) solutions are essential for building trust in AI technologies and their integration in real manufacturing lines. However, most existing methods are tailored to technical experts, limiting their accessibility to diverse user groups such as blue-collar workers in manufacturing lines who use AI for quality control. In this work, we introduce an XAI interface for object detection in industrial manufacturing based on a fine-tuned vision-language model, designed to generate intuitive explanations for non-expert users. We benchmark existing vision-language models and demonstrate that out-of-the-box models often fall short in delivering clear, context-relevant explanations for non-expert users. To address this, we fine-tune a vision-language model and integrate it into our interface, enabling contextualized, accessible explanations for non-expert users. We demonstrate improvements in explanation clarity, instruction adherence, image groundedness, and contextual awareness over GPT 4o-mini on proprietary and public robotics dataset. This approach advances the accessibility and usability of AI explanations, making them more intuitive and applicable in manufacturing domain.

Explainable Object Detection

See all 8 papers

Using Large Language Models and Knowledge Graphs to Improve the Interpretability of Machine Learning Models in Manufacturing
Apr 17, 2026Thomas Bayer, Alexander Lohr, Sarah Weiß +2Manufacturing LLM Grounding
University of Applied Sciences Ravensburg-Weingarten, Germany
Answer-Conditioned Chain-of-Thought Distillation for Few-Shot Industrial Vision with Small VLMs
Jul 12, 2026Shubham RaoCoT Distillation VLM Adaptation
Entropy AI Research Labs Private Limited
PONTE: Personalized Orchestration for Natural Language Trustworthy Explanations
Mar 6, 2026Vittoria Vineis, Matteo Silvestri, Lorenzo Antonelli +2Explainable Artificial Intelligence Faithfulness of Language Model Explanations
Sapienza University of Rome · Syllotips · TellmewAI s.r.l.

Apr 17, 2026J/K move · Enter open · S save

Using Large Language Models and Knowledge Graphs to Improve the Interpretability of Machine Learning Models in Manufacturing

Thomas Bayer, Alexander Lohr, Sarah Weiß, Bernd Michelberger+1

University of Applied Sciences Ravensburg-Weingarten, Germany

Manufacturing LLM Grounding Explainable Artificial Intelligence Explainability Evaluation

Open paper AskPDF arXiv

Explaining Machine Learning (ML) results in a transparent and user-friendly manner remains a challenging task of Explainable Artificial Intelligence (XAI). In this paper, we present a method to enhance the interpretability of ML models by using a Knowledge Graph (KG). We store domain-specific data along with ML results and their corresponding explanations, establishing a structured connection between domain knowledge and ML insights. To make these insights accessible to users, we designed a selective retrieval method in which relevant triplets are extracted from the KG and processed by a Large Language Model (LLM) to generate user-friendly explanations of ML results. We evaluated our method in a manufacturing environment using the XAI Question Bank. Beyond standard questions, we introduce more complex, tailored questions that highlight the strengths of our approach. We evaluated 33 questions, analyzing responses using quantitative metrics such as accuracy and consistency, as well as qualitative ones such as clarity and usefulness. Our contribution is both theoretical and practical: from a theoretical perspective, we present a novel approach for effectively enabling LLMs to dynamically access a KG in order to improve the explainability of ML results. From a practical perspective, we provide empirical evidence showing that such explanations can be successfully applied in real-world manufacturing environments, supporting better decision-making in manufacturing processes.

Abstract

Explainable Object Detection

Explore similar work

Using Large Language Models and Knowledge Graphs to Improve the Interpretability of Machine Learning Models in Manufacturing

Answer-Conditioned Chain-of-Thought Distillation for Few-Shot Industrial Vision with Small VLMs

PONTE: Personalized Orchestration for Natural Language Trustworthy Explanations