SimXAI: A Simple Drop-in Layer Increases Interaction Quality and Understanding in Conversational XAI
Authors: Van Bach Nguyen, Qianli Wang, Nils Feldhus, Jörg Schlötterer, Christin Seifert
Organizations: University of Marburg · Technische Universität Berlin · German Research Center for Artificial Intelligence (DFKI) · University of Groningen
Conversational explainable AI (ConvXAI) agents often use GUS-style architectures with separate modules for intent recognition, dialogue management, and template-based response generation. While faithful to the underlying XAI methods, these pipelines are hard to adapt and often struggle with context-dependent follow-up questions. We present SimXAI, a lightweight LLM-centric conversational layer that replaces these modules with a single tool-calling LLM that bridges access to existing XAI resources. Rather than a standalone agent, SimXAI serves as a modular interface that enables conversational access to external XAI resources. We further introduce an evaluation suite for single- and multi-turn parsing, faithfulness, and contextualization. Across existing and new datasets, SimXAI achieves strong function-selection accuracy and high faithfulness while better handling context-dependent requests. In a user study replacing only the conversational layer of TalkToModel, a representative GUS-based agent, users completed tasks more accurately and in fewer turn. Overall, SimXAI improves GUS-based design while preserving faithfulness and improving conversational quality.
Figures & tables
Figure 1: Comparison of TalkToModel ’s traditional GUS architecture with SimXAI ’s LLM-centric conversational component. Traditional systems rely on multiple components that require rules, templates, or fine-tuning. In contrast, SimXAI requires only function specifications and a single LLM. Both architectures share the same XAI resources.
System
Contextualization
Scalability
Generalization
Response faithfulness
TalkToModel Slack et al. (2023)
Medium
Low (FT, D, T, R)
Low
High
InterroLang Feldhus et al. (2023)
Weak
Low (FT, D, T, R)
Low
High
XAgent Nguyen et al. (2023)
Weak
Medium (FT, D, T, R)
Low
High
SimXAI
Strong
High
High
High
Table 1: Design-level comparison of SimXAI with the conversational layers of prior GUS-based ConvXAI agents in terms of contextualization, scalability, generalization, and response-to-resource faithfulness. The table summarizes architectural properties from the respective system designs. Scalability constraints include: Finetuning (FT), Handcrafted Data (D), Templates (T), and Rules (R).
Function
Description
get_prediction(instance)
Return the model prediction for an instance.
get_global_explanation()
Provide a global explanation of model behavior.
get_local_explanation(instance, prediction_label)
Explain why the model produced a specific prediction.
explain_why_not(instance, expected_label)
Explain why an expected label was not predicted.
explain_how_to_be_that(instance, target_label)
Suggest how the input could be changed to obtain a target label.
explain_how_to_still_be_this(instance)
Identify changes that would preserve the current prediction.
Table 2: Core functions used by SimXAI .
Dataset
#Dialogues
#Turns
#Queries
CoXQL + Compass
–
1
462
Compass-MultiTurn
10
4–6
26
SYN-Sentiment
4
12–24
40
SYN-HeartRate
4
12–24
31
Table 3: Statistics for parsing datasets with increasing conversational complexity.
Aspect
Subtask
What is evaluated
Dataset
SimXAI
Parsing
Single-turn
Correct function selection from a single user query
CoXQL , Compass
86%
Multi-turn
Correct function selection from short dialogue history
Compass-MultiTurn
88%
Dialogue-level
Correct function selection from full conversations with long-range dependencies
SYN-Sentiment
92.5%
SYN-HeartRate
96%
Faithfulness
–
Whether the response is fully supported by the underlying XAI output
SYN-QA-Sentiment
93.6%
–
SYN-QA-HeartRate
91.5%
Table 4: Overview of the evaluation suite for the ConvXAI layer and the corresponding accuracy results of SimXAI .
Figure 3: Agreement between LLM-as-a-judge and human annotation.
Measure
TalkToModel
SimXAI
Accuracy ↑
59.0
92.1
Chat turns ↓
12.4
10.0
Chat time (min) ↓
7.4
5.8
Turns per correct answer ↓
2.70
1.32
Table 5: Objective results from the user study.
BUS-11 item (shortened)
TalkToModel
SimXAI
Interaction Clarity
3.33
4.60
Context Awareness
3.13
4.37
Response Comprehensibility
3.60
4.57
Goal Facilitation
3.10
4.37
Information Sufficiency
3.23
4.37
Information Relevance
3.13
4.13
Table 6: Mean BUS-11 ratings from the user study (1 = strongly disagree, 5 = strongly agree). We report eight relevant items in shortened form, full results are in Table 11 . Cronbach’s α>0.8 indicates high internal consistency.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: CoXQL examples.
Figure 6: Comparison between single-turn and multi-turn parsing.
Agent
Speaker
Original
SimXAI
TalkToModel
User
What if the glucose was 50?
Agent
with the glucose level set to 50, it is still likely to have no diabetes
with the glucose level set to 50, it is still likely to have no diabetes.
User
What if it was 105?
Agent
… glucose is set to 50.0 and bmi is set to 105.0 is predicted likely to have diabetes.
with the glucose level set to 105, it is still likely to have no diabetes.
Interrolang
User
What is the prediction for instance ID 81?
Agent
The instance with id equal to 81 is predicted True.
The instance with id 81 is predicted True.
Appendix
Table 7: Qualitative results
Task
Type
Question Template
Explore the dataset and answer the following question(s).
Data Exploration (global)
• Number of features in the dataset is: • Is the feature <feature> included in the dataset? • The maximum value of <feature> recorded in the dataset is: • Find the minimum value of <feature> recorded in the dataset and choose the correct value:
Explore patient with ID <patient_id> and answer the following question(s).
Data Exploration (local)
• Find the value of <feature_A> for this patient and choose the correct value. • What is the value of <feature_A> for this patient: • How about the <feature_B> value of this patient? • How about this patient’s <feature_B> value?
Find the prediction of the model for patient with ID <patient_id> and answer the following question(s).
Local Explanation
• What does the model predict for this patient? • The most important feature for the prediction is: • The least important feature for the prediction is: • Is <feature> used for the prediction for this patient? • Top 3 most important features for the prediction are:
Counterfactual
• What would be the prediction if the value of <feature> is increased to <value> ? • What would be the prediction if the value of <feature> is decreased to <value> ? • What would be the prediction if it is decreased/increased to <value> ?
Appendix
Table 8: Tasks and corresponding questions used in the user study.
Comparison item
TalkToModel
SimXAI
Overall preference
5 (16.7%)
24 (80.0%)
No preference
1 (3.3%)
Task effectiveness
5 (16.7%)
24 (80.0%)
Equally helpful
1 (3.3%)
Appendix
Table 9: Pairwise comparison results. Most participants preferred SimXAI and judged it more effective.
Metric
TalkToModel
SimXAI
Participants
30
30
Accuracy (unable = wrong)
53.3 ± 19.6
84.1 ± 17.7
Accuracy (excluding unable)
59.0 ± 21.3
92.1 ± 11.6
Total correct / answered (incl. unable)
138 / 262
227 / 270
Total correct / answered (excl. unable)
138 / 238
227 / 246
Unable-to-answer count
24
24
Appendix
Table 10: Objective user-study results. SimXAI improves accuracy and reduces interaction cost. Means are reported with standard deviations.
BUS-11 item
Short Form
TalkToModel
SimXAI
Q1 Chatbot function was easily detectable
Interaction Clarity
3.80 ± 0.76
4.33 ± 0.76
Q2 Easy to find information via the chatbot
Information Sufficiency
4.13 ± 0.78
4.50 ± 0.57
Q3 Communicating with the chatbot was clear
Interaction Clarity
3.33 ± 1.27
4.60 ± 0.56
Q4 Chatbot kept track of conversation context
Context Awareness
3.13 ± 1.31
4.37 ± 0.93
Q5 Responses were easy to understand
Response Comprehensibility
3.60 ± 1.16
4.57 ± 0.68
Q6 Chatbot helped achieve my goal
Goal Facilitation
3.10 ± 1.21
4.37 ± 0.67
Appendix
Table 11: Post-interaction BUS-11 ratings (1–5 Likert). SimXAI is rated higher on 10 of 11 items.
Metric
t
p
Accuracy incl. unable
-9.646
<.001∗∗∗
Accuracy excl. unable
-8.293
<.001∗∗∗
Q1 Chatbot function was easily detectable
-2.898
.0071 ∗∗
Q2 Easy to find information via the chatbot
-2.483
.0190 ∗
Q3 Communicating with the chatbot was clear
-5.091
<.001∗∗∗
Q4 Chatbot kept track of conversation context
-4.570
<.001∗∗∗
Appendix
Table 12: Paired two-tailed t -tests.
Figure 7: General instructions and consent form (identifying information removed for blind submission).
Figure 8: Information on the study procedure and how to conduct tasks.
Figure 9: User interface for conversational XAI agent interaction and question answering.