Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents
Organizations: Amazon, New York, NY, USA · Amazon, Washington, DC, USA · Amazon, Austin, TX, USA
Abstract
Deploying LLM-based analytics agents in enterprise settings requires evaluation frameworks that can reliably detect failures across complex, multi-tool workflows. We present a three-phase comparative study of three evaluation frameworks, each representing a distinct evaluation paradigm (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics), applied to two analytics agents in a controlled research setting using frozen execution traces. Phase 1 quantifies evaluation harness stability through repeated measurements, finding that 97% of configurations meet the stability threshold and that stability patterns are agent-independent. Phase 2 measures known risk detection on 72 ground-truth test cases, where the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms. Phase 3 evaluates unknown risk discovery, uncovering 24 novel failures with zero overlap between functional testing and security red-teaming. Our results demonstrate that (1) the information available to the evaluation harness determines evaluation quality, (2) functional and security evaluation are complementary, and (3) LLM-based evaluation harnesses surface subtle errors that human reviewers miss. We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.
Figures & tables
| Requirement | Strands Evals | PromptFoo | Agenta |
|---|---|---|---|
| Groundedness | FaithfulnessEval (trace) | context-faithfulness (text) | Token overlap |
| Intent | ResponseRelevanceEval (trace) | g-eval (goal rubric) | SequenceMatcher |
| Accuracy | OutputEval (rubric) | g-eval (accuracy rubric) | Levenshtein dist. |
| Tool correct. | TrajectoryEval (trajectory) | llm-rubric (tool names) | Set comparison |
| SQL correct. | OutputEval + SQL from trace | llm-rubric + SQL injection | Regex matching |
| Framework | Configs | Stable ( ) | Avg Std | Avg DC |
|---|---|---|---|---|
| Agenta | 5 | 5/5 (100%) | 0.000 | 100% |
| PromptFoo | 12 | 12/12 (100%) | 0.025 | 92% |
| Strands | 12 | 11/12 (92%) | 0.053 | 82% |
| Framework | Avg Std | Avg DC | Evaluation Depth |
|---|---|---|---|
| Agenta | 0.000 | 100% | Deterministic (string matching) |
| PromptFoo | 0.025 | 92% | LLM judge (text-based) |
| Strands | 0.053 | 82% | LLM judge (full trace) |
| Dimension | Agent A | Agent B | Combined |
|---|---|---|---|
| SQL correctness | 14 | 14 | 28 |
| Tool correctness | 4 | 5 | 9 |
| Accuracy | 1 | 5 | 6 |
| Intent | 1 | 3 | 4 |
| Groundedness | 0 | 2 | 2 |
| Framework | TP | FP | FN | TN | Prec | Rec | F1 | FPR |
|---|---|---|---|---|---|---|---|---|
| Strands | 30 | 4 | 3 | 35 | 88.2% | 90.9% | 0.896 | 10.3% |
| PromptFoo | 29 | 19 | 4 | 20 | 60.4% | 87.9% | 0.716 | 48.7% |
| Agenta | 33 | 39 | 0 | 0 | 45.8% | 100% | 0.629 | 100% |
| Metric | Strands | PromptFoo | Agenta |
|---|---|---|---|
| Cases generated | 40 | 90 | 0 |
| Novel failures | 9 | 15 | 0 |
| False discovery rate | 6.1% | 20.8% | 85% |
| Failure surface | Functional | Security | N/A |
| # | Type | Fwk | Cases | Sev. |
|---|---|---|---|---|
| 1 | SUBSTR negative index in SQL | Strands | 6 | High |
| 2 | Missing count_result_rows | Strands | 1 | Med |
| 3 | Retry storm / tool explosion | Strands | 1 | Med |
| 4 | Schema fabrication | Strands | 1 | Med |
| 5 | Schema metadata leakage | PromptFoo | 10 | High |
| 6 | Storage path/resource exposure | PromptFoo | 5 | High |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Judge | Evaluator | Avg Within-Case Std | Decision Consist. | ||
|---|---|---|---|---|---|
| Agent A | Agent B | Agent A | Agent B | ||
| Claude Haiku | faithfulness | 0.080 | 0.063 | 80% | 85% |
| Claude Haiku | goal | 0.022 | 0.006 | 95% | 100% |
| Claude Haiku | style | 0.071 | 0.110 | 85% | 55% |
| Claude Haiku | tool_sel. | 0.034 | 0.066 | 94% | 87% |
| Sonnet 4.6 | faithfulness | 0.009 | 0.012 | 100% | 100% |
| Framework | TP | FP | FN | TN | Prec | Rec | F1 | FPR |
|---|---|---|---|---|---|---|---|---|
| Strands | 15 | 1 | 2 | 18 | 93.8% | 88.2% | 0.909 | 5.3% |
| PromptFoo | 15 | 8 | 2 | 11 | 65.2% | 88.2% | 0.750 | 42.1% |
| Agenta | 17 | 19 | 0 | 0 | 47.2% | 100% | 0.642 | 100% |
| Framework | TP | FP | FN | TN | Prec | Rec | F1 | FPR |
|---|---|---|---|---|---|---|---|---|
| Strands | 15 | 3 | 1 | 17 | 83.3% | 93.8% | 0.882 | 15.0% |
| PromptFoo | 14 | 11 | 2 | 9 | 56.0% | 87.5% | 0.683 | 55.0% |
| Agenta | 16 | 20 | 0 | 0 | 44.4% | 100% | 0.615 | 100% |
| Evaluator | Recommended Judge | Stability |
|---|---|---|
| faithfulness | Claude Haiku | EXCELLENT |
| goal | Claude Haiku | DETERMINISTIC |
| style | Claude Haiku | DETERMINISTIC |
| tool_selection | Claude Haiku | DETERMINISTIC |
| Evaluator | Recommended Judge | Stability |
|---|---|---|
| faithfulness | Claude Sonnet 4.6 | EXCELLENT |
| goal | Claude Sonnet 4.6 | EXCELLENT |
| style | Haiku / Sonnet 4.6 | GOOD |
| tool_selection | Claude Haiku | EXCELLENT |
| Phase | Framework | Agent | Evals |
|---|---|---|---|
| Phase 1 | Agenta | Agent A | 460 |
| Phase 1 | Agenta | Agent B | 375 |
| Phase 1 | PromptFoo | Agent A | 1,140 |
| Phase 1 | PromptFoo | Agent B | 1,125 |
| Phase 1 | Strands | Agent A | 1,380 |
| Phase 1 | Strands | Agent B | 1,125 |
| Judge | Model | Temp | Tokens | Notes |
|---|---|---|---|---|
| Claude Haiku | claude-haiku-4.5 | Def. | Def. | Most stable judge across both frameworks |
| Claude Sonnet 4.6 | claude-sonnet-4.6 | Def. | Def. | Best balance of stability and depth in Strands |
| Qwen3 32B | qwen3-32b | Def. | Def. | Thinking tags break structured output validation |
| Judge | Model | Temp | Tokens | Notes |
|---|---|---|---|---|
| Claude Haiku | claude-haiku-4.5 | 0 | 4,096 | Deterministic on all 4 evaluation harnesses |
| Claude Sonnet 4.6 | claude-sonnet-4.6 | 0 | 4,096 | Excellent stability on most evaluation harnesses |
| Qwen3 32B | qwen3-32b | 0 | 4,096 | showThinking: false required |
| GPT-OSS-120B | gpt-oss-120b | 0 | 4,096 | Tested but incompatible (reasoning tags break parser) |
| MiniMax M2 | minimax-m2 | Def. | 4,096 | Tested but incompatible (thinking tags break parser) |
| Parameter | Value |
|---|---|
| Model | Claude Sonnet 4.5 |
| Temperature | Default (not explicitly set) |
| Region | us-east-1 |
| Memory | AgentCore Memory (session-based) |
| Framework | Type | Setup | Sel.? |
|---|---|---|---|
| Strands Evals | LLM (trace) | 30 min | Yes |
| PromptFoo | LLM (text) + red-team | 30 min | Yes |
| Agenta | Deterministic | 2 hrs | Yes |
| DeepEval | LLM (trace) | 4 hrs | No |
| Ragas | LLM (RAG) | 4–5 hrs | No |
| TruLens | LLM (instrumented) | 3 hrs | No |
| Tool | Purpose |
|---|---|
| sql_query | SQL queries against the data warehouse |
| read_query_results | Read query results from cloud storage |
| count_result_rows | Count rows in result files |
| calculator | Arithmetic calculations |
| generate_graph | Visualizations |
| resolve_date_range | Date resolution |
| Tool | Purpose |
|---|---|
| sql_query | SQL queries against the data warehouse |
| count_result_rows | Count rows in result files |
| sample_results | Sample data from result files |
| code_interpreter | Complex metric calculations via code interpreter |
| generate_graph | Visualizations |
| resolve_date_range | Date resolution |
| Judge Model | Avg Score | Corr. w/ Sonnet | Latency | Cost |
|---|---|---|---|---|
| Claude Haiku | 0.82 | 3s | 1 | |
| Claude Sonnet 4.6 | 0.79 | — | 8s | 5 |
| Claude Opus 4 | 0.78 | 28s | 25 |
| Category | Text-Only LLM Judging | Trace-Based LLM Judging |
|---|---|---|
| Concept Explanation | 0.176 | 0.230 |
| Comparison | 0.195 | 0.397 |
| Visualization | 0.214 | 0.358 |
| Diagnostic | 0.221 | 0.348 |
| Data Retrieval | 0.230 | 0.353 |
| Accuracy Analysis | 0.238 | 0.407 |
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,307 papers