LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Token Counts Are Not Model Lineage: A Frozen-Threshold Holdout Study of Black-Box LLM API Fingerprinting
Aug 30, 2026Bo ChenLLM Evaluation Language Model Fingerprinting
Institute of Computing Technology, Chinese Academy of Sciences
Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation
Aug 30, 2026Nishant Balepur, Paiheng Xu, Wei Ai +3LLM Evaluation Multiple-Choice Question Answering
University of Maryland · New York University · Nanyang Technological University
Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments
Aug 30, 2026Lin Chen, Yitong Chen, Yong LiLLM Evaluation Belief Updating in LLMs
1Network Science Institute, Northeastern University · Department of Physics, Northeastern University · Department of Electronic Engineering, BNRist, Tsinghua University
Evaluating the Capabilities of LLMs for Persuasive Dialogue
Aug 30, 2026Jordan Robinson, Angus R. Williams, Katie Atkinson +1LLM Evaluation Multi-Agent Debate
University of Liverpool · 2The Alan Turing Institute · University of Leeds
Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
Aug 25, 2026Xiulin Yang, Ethan Gotlieb Wilcox, Catherine ArnettMultilingual Language Model Evaluation LLM Evaluation
Georgetown University · EleutherAI
Beyond Semantic Accuracy: Consequence-Aware Evaluation for Safety-Critical Language Understanding
Aug 25, 2026Yujing Chang, Thinh Pham, Van-Phat Thai +4LLM Safety Benchmarks LLM Evaluation
1ATMRI, Nanyang Technological University (NTU), Singapore · Centre of AI Research, VinUniversity, Vietnam · School of Management, Indian Institute of Technology Mandi, India
LongWoF-Bench: Evaluating EvoMap Genes for Verifiable Long-Workflow Tasks
Aug 24, 2026Xiao Zhang, Qumeng Sun, Jiahao Li +3Agent Memory LLM Evaluation
Infinite Evolution Lab, EvoMap
ExecRubrics: Executable Tool-Augmented Rubrics for Verifiable and Efficient Long-Form Evaluation
Aug 23, 2026Kaustubh D. Dhole, Charles L. A. Clarke, Eugene Y. AgichteinLLM Evaluation Rubric-Based Evaluation
Department of Computer Science Emory University Atlanta, USA · School of Computer Science University of Waterloo Waterloo, Canada
FlavourBench: Executable Culinary Reward Maps for Language Model Evaluation and Post-Training
Aug 20, 2026Josef Chen, Erim HayretciLLM Evaluation Supervised Fine-Tuning
Independent Researcher · Imperial College London
FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models
Aug 20, 2026Dingzirui Wang, Xuanliang Zhang, Keyan Xu +2LLM Evaluation Automated Theorem Proving
Harbin Institute of Technology
Decomposing Wrong-Consensus Agreement in LLM Self-Consistency
Aug 19, 2026Lizhuo Zhang, Mengmeng Tang, Chenfeng Long +2LLM Evaluation LLM Reliability
College of Information and Intelligence, Hunan Agricultural University, Changsha 410128, China · Yuelushan Laboratory, Changsha 410128, China · School of Computer Science and Technology, Changsha University of Science and Technology, Changsha 410114, China +1
The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations
Aug 18, 2026Emma Yanyang Kong, JJ Tan, Ishan Gupta +8LLM Evaluation LLM-as-a-Judge
Netflix · Los Gatos, California, USA
TokEval: A Tokenizer Evaluation Suite
Aug 18, 2026Clara MeisterLLM Evaluation
EPFL
PTXBench: Benchmarking and Adapting LLMs for GPU Kernel Optimization with Architecture-specific PTX
Aug 18, 2026Genghan Zhang, Yixin Dong, Chengze Fan +4LLM Evaluation GPU Kernel Optimization
Stanford University · Carnegie Mellon University · Independent Researcher +1
A Probe Direction Is a Property of Its Prompt
Aug 13, 2026Valentin NoëlPrompt Sensitivity LLM Evaluation
Devoteam
Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation
Aug 13, 2026Junhao Luo, Ning Huang, Ziqi Sha +2LLM Evaluation LLM Auditing
School of Statistics and Data Science, Southwestern University of Finance and Economics · School of Statistics and Data Science, Southwestern University of Finance and Economics Chengdu, China
TsuGO: Probing Search Efficiency in LLM Reasoning via Go Life-and-Death Problems
Aug 13, 2026Shunwen Bai, Ziping Ma, Chaoyang Zhang +4LLM Evaluation Inference-Time Search
Zhejiang University · Ant Group · Central South University
Which LLM Is Your Ideal Companion? Evaluating Emotional Companion Capabilities of LLMs Based on Adult Attachment Theory
Aug 13, 2026Junkai Zhou, Shiting Guan, Zhaoyi ZhangLLM Evaluation Personality Modeling in Language Models
LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation
Aug 13, 2026Chenrun Wang, Mingxuan Zhu, Tiancheng Huang +6LLM Evaluation Benchmark Design
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 2Shanghai Innovation Institution, Shanghai, China
The Embedder's Dilemma: LLMs Are Better, but at What Cost?
Aug 13, 2026Adnan El Assadi, Niklas Muennighoff, Jinhyuk LeeLLM Inference Efficiency LLM Evaluation
Harvard University · Stanford University · Independent Researcher
Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence
Aug 12, 2026Justin Zhao, Himaghna Bhattacharjee, Hannah Korevaar +2LLM Evaluation LLM-as-a-Judge
1Meta Superintelligence Labs · 2FAIR at Meta
Novels generated by language models show compressed formal variation
Aug 12, 2026Mehdy Sedaghat Payam, Justin QuinnLLM Evaluation Long-Form Document Generation
University of Maryland · University of West Bohemia · Charles University
NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation
Aug 12, 2026Jiarui Ma, Jianghan Wang, Yuheng Ma +2LLM Evaluation Benchmark Design
Southern University of Science and Technology School of Microelectronics Shenzhen, China
Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
Aug 12, 2026Rodrigo Guedes de Souza, Alison R. PanissonLLM Evaluation Language Model Generation Evaluation
Federal University of Santa Catarina (UFSC) Brazil
RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks
Aug 12, 2026Jinjun Huang, Zhongzhen Wen, Tongtong Xu +3LLM Evaluation GPU Kernel Generation
College of Computer Science and Technology and the State Key Laboratory of Blockchain and Data Security, Zhejiang University · State Key Lab for Novel Software Technology, Nanjing University · Software Engineering Application Technology Laboratory, Huawei +1
Accuracy and Order Sensitivity Diverge Under Label-Free Strategies
Aug 12, 2026Karl Hanna, Chen FengLLM Evaluation Multiple-Choice Question Answering
Queen’s University Belfast
OEIS Open: How many conjectures can language models turn into theorems?
Aug 12, 2026Tom AdamczewskiLLM Evaluation Automated Theorem Proving
Epoch AI
Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfaction
Aug 12, 2026Mariya I. VasilevaLLM Evaluation Instruction Following
Work done while at Meta Superintelligence Labs.
Benchmark-Based Comparative Assessment of Publicly Benchmarked Indian Foundation Models: A Capability and Evaluation-Maturity Framework
Aug 12, 2026Avinash Agarwal, Vridhi JainLLM Evaluation
Unique Identification Authority of India, New Delhi, India
The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance
Aug 12, 2026Shailja Thakur, Sungeun An, Chad DeLuca +1Prompt Sensitivity LLM Evaluation
IBM Research India · IBM Research Almaden

Aug 30, 2026, cs.CLJ/K move · Enter open · S save

Token Counts Are Not Model Lineage: A Frozen-Threshold Holdout Study of Black-Box LLM API Fingerprinting

Bo Chen

Institute of Computing Technology, Chinese Academy of Sciences

LLM Evaluation Language Model Fingerprinting

Open paper AskPDF arXiv

Black-box model attribution is increasingly relevant when large language models (LLMs) are served through relay and reseller APIs. A tempting low-cost signal is the prompt-token count returned by an OpenAI-compatible endpoint: two models that share a tokenizer and chat template may produce the same count sequence up to a fixed offset. Yet the validity of this signal for broader \emph{model-family} attribution has received little direct holdout testing. We conduct a frozen-threshold study over 24 labeled endpoint pairs, split evenly into a development set and an untouched holdout set, with three temporal repeats and 30 controlled texts per pair. We introduce a validity-gated result contract that distinguishes an observed dissimilarity from an uninformative measurement caused by missing usage data, rate limits, or endpoint policy. The resulting shift-invariant exact-match score perfectly separates the 12 development pairs, yielding a frozen threshold of 0.725. On holdout, however, only 6 of 12 pairs are eligible under the pre-specified three-repeat rule. Among eligible pairs, balanced accuracy is 0.75, sensitivity is 0.50 (95% Wilson interval 0.15--0.85), and specificity is 1.00 (0.342--1.00). Two same-family pairs---Qwen 3.8 and DeepSeek V4 variants---fall below the frozen threshold. Across 4,320 formal API calls, every log is replayable, while holdout contains 189 non-200 responses and 157 successful responses without prompt-token usage. The study therefore validates token-count consistency as a fingerprint of a shared \emph{tokenization stack}, but rejects its use as a standalone necessary test for model-family lineage.

Momentum

Related topics

Latest papers 1,643