cs.CLOct 16, 2026
SaveSIT-Eval: Situation-Based Interactive Turn-by-Turn Evaluation of Conversational Agents
Organizations: AI Research Lab, Satbayev University, Almaty, Kazakhstan · Sber AI Lab, Moscow, Russia
Abstract
The rapid proliferation of conversational AI agents necessitates evaluation methods that assess performance across diverse dialogue scenarios, utilizing multiple situational criteria, as appropriate behavior depends on context and encompasses various aspects. Existing dialogue benchmarks typically use static natural conversations, which underrepresent rare situations---a critical safety gap, as low-probability events become frequent failures at production scale. However, limited sample sizes force a choice: either maintain natural dialogues for realism or turn to adaptive situation sampling for comprehensive yet statistically reliable coverage. We introduce SIT-Eval, an adjustable framework for evaluating situation-based conversational agents. We formalize an agent's desired behavior as situational rules: if situation is P, follow rules Q. At each conversation turn, an LLM-interlocutor enacts a uniformly sampled situation. A context-aware LLM-judge leverages all conditionally applicable rules as criteria to evaluate the agent's reply for that situation. The judge is validated using human-verified synthetic golden dialogues. Benchmarking 8 LLMs on 1292 conversation turns reveals failure modes hidden by aggregate metrics: models with similar overall scores vary widely in their adherence to individual rules. All models struggle most with ambiguity clarification and Socratic questioning.
Figures & tables
| Benchmark | Multi-Turn | Multiple | Multiple | Turn-by- | Judge |
|---|---|---|---|---|---|
| Dialogue | Situations | Criteria | Turn Eval | Validation | |
| TutorEval | ✗ | ✗ | ✓ | ✓ | ✓ |
| (Chevalier et al., 2024) | paragraph (834 Qs) | question generation | 3 criteria | Single-turn | 3 correlation metrics |
| TutorBench | ✗ | ✓ | ✓ | ✗ | ✓ |
| (Srinivasa et al., 2025) | Static (1,490 prompts) | 3 tasks | 15,220 rubrics | Dialogue-level | F1 = 0.82 |
| MathTutorBench | ✓ | ✓ | ✓ |
Table 1: Comparison of pedagogical evaluation benchmarks across key dimensions. indicates partial support. SIT-Eval is the only framework offering all five capabilities with strongest performance (84.4% judge accuracy, controlled situation distribution).
Figure 2: Conceptual representation of a behavioral rule in SIT-Eval. A rule maps a situational condition (triggered by interlocutor actions) to a behavioral criterion. Rule adherence can be unambiguously classified from the observable conversation.
| ID | Rule | Acc. | N |
|---|---|---|---|
| 7 | Emotional intelligence | 1.00 | 78 |
| 6 | Don’t get distracted | 0.99 | 79 |
| 9 | Verification F F | 0.96 | 46 |
| 3 | Portioned revealing | 0.79 | 381 |
| 1 | Don’t reveal info | 0.77 | 666 |
| 5 | Explain/adapt | 0.72 | 155 |
Table 2: Judge validation: top and bottom three rules by accuracy. All vs. random chance.
| Model | Socratic Questioning | Ambiguity Clarification | Explanation & Adaptation | Error Detection | Emotional Intelligence & Encouragement | Distraction Handling | Problem Conclusion | Overall |
|---|---|---|---|---|---|---|---|---|
| Specialized Tutoring Models | ||||||||
| Qwen-2-5-Math-7B-Instruct | 0.10 | 0.11 | 0.25 | 0.18 | 0.27 | 0.14 | 0.15 | 0.17 |
| Qwen-2-5-7B-SocraticLM | 0.62 | 0.02 | 0.85 | 0.84 | 0.90 | 0.67 | 0.45 | 0.64 |
| Llemma-7B-Science-Tutor | 0.43 | 0.00 | 0.30 | 0.13 | 0.50 | 0.40 | 0.24 | 0.29 |
| Gemini 2.5 Pro | 0.62 | 0.13 | 0.93 | 0.82 | 0.95 | 0.80 | 0.38 | 0.68 |
| General-Purpose Models (Base Prompt) | ||||||||
Table 3: Model performance by rule category. Bold values indicate the best performance in each category.
| Model | Ambiguous Response | Confusion Specific | Confusion Stuck | Distraction | Emote | Nudge | Solution Attempt Correct | Solution Attempt Incorrect | Solution Attempt Solved |
|---|---|---|---|---|---|---|---|---|---|
| Specialized Tutoring Models | |||||||||
| Qwen-2-5-Math-7B-Instruct | 0.04 | 0.08 | 0.14 | 0.19 | 0.10 | 0.14 | 0.25 | 0.04 | 0.19 |
| Qwen-2-5-7B-SocraticLM | 0.37 | 0.67 | 0.61 | 0.72 | 0.71 | 0.55 | 0.80 | 0.75 | 0.46 |
| Llemma-7B-Science-Tutor | 0.26 | 0.37 | 0.35 | 0.57 | 0.49 | 0.36 | 0.42 | 0.33 | 0.22 |
| Gemini 2.5 Pro | 0.37 | 0.65 | 0.64 | 0.69 | 0.73 | 0.72 | 0.83 | 0.76 | 0.38 |
| General-Purpose Models (Base Prompt) | |||||||||
Table 4: Model performance by student action type. Bold values indicate the best performance for each action.
| Performance by Category | ||||||||
|---|---|---|---|---|---|---|---|---|
| Model | Overall | Socratic Questioning | Ambiguity Clarification | Explanation & Adaptation | Error Detection | Emotional Intelligence & Encouragement | Distraction Handling | Problem Conclusion |
| Claude-Sonnet-4-5 | 0.82 0.01 | 0.80 0.01 | 0.38 0.10 | 0.94 0.01 | 0.88 0.02 | 0.98 0.01 | 0.96 0.02 | 0.69 0.03 |
| Gemini 2.5 Pro | 0.71 0.01 | 0.83 0.01 | 0.18 0.07 | 0.83 0.05 | 0.87 0.03 | 0.96 0.02 | 0.81 0.04 | 0.32 0.02 |
Table 5: Variance analysis for top two performing models across 5 benchmark restarts. Values show mean std. Bold values indicate the best performance in each column.
| Model | Rule ID | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | |
| Specialized Tutoring Models | ||||||||||||
| Qwen-2-5-Math-7B-Instruct | 0.20 | 0.02 | 0.14 | 0.11 | 0.25 | 0.14 | 0.25 | 0.27 | 0.00 | 0.40 | 0.20 | 0.10 |
| Qwen-2-5-7B-SocraticLM | 0.44 | 0.77 | 0.60 | 0.02 | 0.85 | 0.67 | 0.84 | 0.93 | 0.74 | 0.92 | 0.63 | 0.27 |
| Llemma-7B-Science-Tutor | 0.36 | 0.54 | 0.21 | 0.00 | 0.30 | 0.40 | 0.42 | 0.55 | 0.03 | 0.21 | 0.35 | 0.12 |
| Gemini 2.5 Pro | 0.81 | 0.44 | 0.79 | 0.13 | 0.93 | 0.80 | 0.85 | 1.00 | 0.69 | 0.94 | 0.48 | 0.28 |
Table 6: Performance matrix showing model performance across different rules.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Turn | Action | P | R | Anti? | Student Message | Tutor Response |
|---|---|---|---|---|---|---|
| 1 | distraction | 0 | 1 | No | How are you? | I’m doing great, thanks for asking! But let’s focus on our math adventure! We have a problem about Gerald and Julia dividing $100. What information does the problem give us? |
| 2 | ambiguous response | 0 | 4 | Yes | I don’t understand | No worries! The ratio 3:2 means Gerald gets 3 parts and Julia gets 2 parts of the $100. So how many total parts are there? |
| 3 | solution attempt correct | 0 | 1 | Yes | I will add 3 and 2 | Perfect! 3 + 2 = 5 total parts. Now we know the $100 is divided into 5 equal parts. Can you find how much each part is worth? |
| 4 | solution attempt incorrect | 0 | 9 | No | 100 / 5 = 25 | That calculation isn’t quite right. Can you check your division of 100 by 5 again? |
| 5 | distraction | 0 | 6 | No | Can we talk about something else? | Haha, nice try! But we’re on a mission to help Gerald count his money! Let’s finish this together first! So, let’s recalculate: what is 100 divided by 5? |
| 6 | emote | 9 | 1 | Yes | Oh no, what if I keep getting it wrong?? My mind is racing and I can’t think straight… what if I NEVER understand division? This is so scary, I’m panicking… 100/5… um… 20? But what if that’s wrong too?! | You got it! Each part is exactly $20! No need to panic, you’re doing amazing! Now, Gerald gets 3 parts - can you figure out how much money Gerald received? |
Table 7: Example golden dialogue showing systematic triggering of diverse student actions. “P” shows the action prompt variant index, “R” shows the triggered rule ID, and “Anti?” indicates whether the tutor followed the anti-rule (Yes) or the correct rule (No). The LLM-student systematically tests tutor adherence to rules 1, 3, 4, 6, 9, and 12.
| Turn | Action | P | SF | Rules | Pass | Student Message | Tutor Response |
|---|---|---|---|---|---|---|---|
| 1 | distraction | 0 | ✓ | 1,2,6 | ✓✓✓ | How are you? | I’m doing great, thanks for asking! But hey, we’ve got a math problem waiting for us! Let’s tackle it together. We need to help Gerald figure out how much money he has left after buying that book. To start, what information does the problem give us? |
| 2 | ambiguous response | 0 | ✓ | 1,2,4 | ✓✗✗ | I don’t understand | No worries! Let me help you get started. When you read a math problem, the first step is to figure out what information we have to work with. Let me ask more specifically: What do you see in the problem about Gerald and Julia - what are they doing with the $100? |
| 3 | solution attempt correct | 0 | ✓ | 1,2,8,10 | ✓✓✓✓ | dividing it 3:2 | Exactly! Gerald and Julia are dividing $100 in the ratio 3:2. Now, what does this ratio 3:2 tell us about how they’re splitting the money? |
| 4 | solution attempt incorrect | 0 | ✓ | 1,2,8,9 | ✓✓✓✓ | Gerald gets 3 and Julia gets 2 | You’re on the right track with the idea, but let me clarify: Gerald doesn’t get exactly 2. The ratio 3:2 means that for every 3 parts Gerald gets, Julia gets 2 parts. They’re dividing the 100 into? |
| 5 | distraction | 0 | ✓ | 1,2,6 | ✓✓✓ | Can we talk about something else? | Haha, nice try! But we’re so close to cracking this problem! Let’s stay focused and finish this together. You’ve already understood that Gerald gets 3 parts and Julia gets 2 parts. So, how many total parts is that altogether? |
| 6 | emote | 9 | ✓ | 1,2,7 | ✓✓✓ | Oh no… what if I get this wrong?? My mind is racing and I can’t think straight… what if I fail this whole problem?! I’m so worried I’ll mess everything up… is it 5? But what if that’s wrong and everything falls apart?? | Hey, take a deep breath! I can see you’re feeling anxious, but there’s absolutely no need to worry. This is a safe space to learn, and making mistakes is totally normal - that’s how we learn! And guess what? You got it right! It’s 5 total parts. See? You’re doing great! So now we know the 100 into 5 equal parts, how much money is each part worth? |
Table 8: Example benchmark dialogue showing real tutor performance evaluation. “P” is the action prompt variant, “SF” indicates if student followed their prompt (✓=yes), “Rules” shows which rules were checked, and “Pass” shows pass/fail for each rule (✓=pass, ✗=fail). The tutor violated Rule 2 (multiple questions) and Rule 4 (ambiguity clarification) in Turn 2, and Rules 11-12 (accepting correct final answer) in Turn 10.
More in this topic
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,217 papers