cs.AINov 9, 2026

Can the Remaining Answers Change the Prediction? Worst-Case Stability Checks for Personalized Rating Models

Authors: Priyanka Siddappa, Muntaqim Ahmed Raju, Shifat Haider Al Amin, Ruizhe Ma

Organizations: University of Massachusetts Lowell, Lowell, MA 01854, USA

Abstract

Missing ratings can overturn a personalized model's current prediction. For completed models with additive response scores, exact worst case bounds check all missing assignments without changing the current predictor. We evaluate this audit retrospectively in two self-triage cohorts, 108 clinicians rating three simulated psychiatric interviews, and 126 counseling demonstrations annotated by ten experts. Under shared random half histories, Fisher adaptation based on observed ratings reduces log loss relative to response frequency by 0.1071 nats in the clinician audit (95% CI [0.0867,0.1261]) and 0.0223 in counseling (95% CI[0.0121,0.0321]). The counseling test excludes target experts and source videos from training. Certification covers 5.4% and 10.5% of targets; fixed width acquisition raises coverage to 13.7% and 28.6% at the same budgets. With weights and responses held fixed, the observed-rating update also has lower loss than an affine partial update while certifying exactly the same evaluated targets. The certificate establishes agreement with the completed model, not correctness or stability at intermediate updates. These rating tasks test the method; clinical usefulness remains untested.

Figures & tables

Explore similar work

CardsList
  1. When Can Old Evaluations Certify a New Model? Label-Efficient Release Decisions under Evaluator Drift

    Sep 26, 2026Joyanta Jyoti Mondal, Mridul Banik, Md. Shifatul Ahsan Apurba +1Sequential Hypothesis TestingLLM Auditing

  2. Hierarchical Bayesian Crowdsourcing with Item Difficulty

    May 29, 2024Seong Woo Han, Ozan Adıgüzel, Bob CarpenterInter-Rater ReliabilityLatent Variable Models