cs.CLNov 4, 2026

Presentation Sensitivity in Typed Decision Models for Subjective Text Evaluation

Authors: Jatin Agrawal, Radhika Mamidi

Organizations: LTRC, IIITH

Abstract

Typed decision models offer structured choices, scores, and probabilities for low-cost automated evaluation. We present a controlled audit of presentation sensitivity in Jev, Clef Flash, and Perplexity Decider across native choice and score interfaces. Across six datasets with 9,000 multiply annotated humor, helpfulness, and summarization items, we collect 702,000 judgments under interventions that vary positions, identifiers, authorship claims, endorsements, decision framing, and rating rubrics while preserving the evaluated texts. Jev's second-position probability advantage reaches 6.11 percentage points on summaries; reversing reader endorsements changes Perplexity's selected humor text on 40.40% of pairs; and rejection framing reduces Clef's summary agreement with human preferences by 23.36 points. Rating formats also shift score levels while often preserving rankings. Across these findings, repeatability, presentation robustness, and human alignment emerge as distinct properties of an evaluator. Reliable use therefore requires validating the decisions and score interpretations on which an evaluation pipeline depends.

Figures & tables

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges

    Oct 13, 2026Fabian Lukassen, Christoph Weisser, Alexander SilbersdorffTyped Decision ModelsBias in LLM-as-a-Judge

  2. How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

    Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM EvaluationLLM-as-a-Judge