LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactions
Apr 30, 2026Zhuoran Pan, Yue Li, Zhi Guan +2Financial Services LLM Evaluation
School of Computer Science Peking University Beijing, PA 100871 · Taiyuan University of Technology Taiyuan, Shanxi, PA 030024 · Peking University Beijing, PA 100871
Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading
Apr 30, 2026Nicholas Sadjoli, Tim Siefken, Atin Ghosh +2LLM Evaluation Language Model Generation Evaluation
SAP · Stanford University
Exploring Applications of Transfer-State Large Language Models: Cognitive Profiling and Socratic AI Tutoring
Apr 30, 2026Minori NoguchiLLM Evaluation Intelligent Tutoring Systems
Hitachi High-Tech Corporation
From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
Apr 30, 2026Qingyu Ren, Tianjun Pan, Xingzhou Chen +1Reward Modeling RL for Language Models
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · 2Shanghai Artificial Intelligence Laboratory
Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models
Apr 30, 2026Renwei Meng, Bowen Zhang, Jian Wang +4LLM Evaluation LLM Auditing
Anhui University, Hefei, China
A Reproducibility Study of LLM-Based Query Reformulation
Apr 30, 2026Amin Bigdeli, Radin Hamidi Rad, Hai Son Le +4LLM Evaluation LLM Prompting
University of Waterloo Waterloo, Ontario, Canada · Mila – Quebec AI Institute Montreal, Quebec, Canada · Toronto Metropolitan University Toronto, Ontario, Canada +2
Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation
Apr 30, 2026Jon-Paul CacioliLLM Evaluation
Independent Researcher, Melbourne, Australia
Measurement Risk in Supervised Financial NLP: Rubric and Metric Sensitivity on JF-ICR
Apr 30, 2026Sidi Chang, Peiying Zhu, Yuxiao Chen +1LLM Evaluation Rubric-Based Evaluation
Blossom AI Labs, Tokyo, Japan · Blossom AI Labs, San Francisco, USA
LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps
Apr 30, 2026Keito Inoshita, Xiaokang Zhou, Akira Kawai +1LLM Evaluation Annotator Disagreement
Faculty of Business and Commerce, Kansai University, Osaka, Japan · Faculty of Business Data Science, Kansai University, Osaka, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan +2
REBENCH: A Procedural, Fair-by-Construction Benchmark for LLMs on Stripped-Binary Types and Names (Extended Version)
Apr 30, 2026Jun Yeon Won, Xin Jin, Shiqing Ma +1LLM Evaluation Software Reverse Engineering
Ohio State University Columbus, OH, United States · University of Massachusetts Amherst Amherst, MA, United States
Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation
Apr 29, 2026Jon-Paul CacioliLLM Evaluation Language Model Generation Evaluation
Independent Researcher, Melbourne, Australia
When Your LLM Reaches End-of-Life: A Framework for Confident Model Migration in Production Systems
Apr 29, 2026Emma Casey, David Roberts, David Sim +1LLM Evaluation
Verint Systems Inc, Melville, NY, USA
CL-bench Life: Can Language Models Learn from Real-Life Context?
Apr 29, 2026Shihan Dou, Yujiong Shen, Chenhao Huang +35LLM Evaluation LLM Grounding
Hunyuan Team, Tencent · Fudan University
Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics
Apr 29, 2026Jatin Bhusal, Nancy Mahatha, Aayush Acharya +1LLM Evaluation Educational Assessment
Research and Incubation Center (RAIN), Sunway College Kathmandu
Do Larger Models Really Win in Drug Discovery? A Benchmark Assessment of Model Scaling in AI-Driven Molecular Property and Activity Prediction
Apr 29, 2026Jinjiang Guo, Sheng DingLLM Evaluation Molecular Property Prediction
Global Health Drug Discovery Institute, Beijing 100084, China · School of Pharmaceutical Sciences, Tsinghua University, Beijing 100084, China
Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization
Apr 29, 2026Yash Ganpat SawantLLM Evaluation LLM Personalization
Independent AI Researcher
Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs
Apr 29, 2026Gilberto Sussumu Hida, Danilo Monteiro Ribeiro, Erika YahataSoftware Engineering LLM Evaluation
AIBL · CESAR School · CMCC +1
A Systematic Comparison of Prompting and Multi-Agent Methods for LLM-based Stance Detection
Apr 29, 2026Genan Dai, Zini Chen, Yi Yang +1Multi-Agent LLM Systems LLM Evaluation
School of Artificial Intelligence, Shenzhen Technology University, Shenzhen, China
TRIP-Evaluate: An Open Multimodal Benchmark for Evaluating Large Models in Transportation
Apr 29, 2026Han Gong, Zhen Zhou, Yunyang Shi +4LLM Evaluation Intelligent Transportation Systems
School of Transportation Southeast University Nanjing, Jiangsu 211189 · School of Artificial Intelligence and Computer Science Jiangnan University Wuxi, Jiangsu 214122 · Department of Civil and Environmental Engineering Hong Kong Polytechnic University Kowloon, Hong Kong
BatteryPass-12K: The First Dataset for the Novel Digital Battery Passport Conformance Task
Apr 28, 2026Tosin Adewumi, Martin Karlsson, Lama Alkhaled +1LLM Evaluation
Machine Learning Group, EISLAB · Research Center for Advanced Battery Technology, Luleå University of Technology, Sweden.
QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits
Apr 28, 2026Zhenxiao Fu, Lei Jiang, Fan ChenLLM Evaluation Code Generation
Indiana University Bloomington
CGU-ILALab at FoodBench-QA 2026: Comparing Traditional and LLM-based Approaches for Recipe Nutrient Estimation
Apr 28, 2026Wei-Chun Chen, Yu-Xuan Chen, I-Fang Chung +1LLM Evaluation Cost-Aware Inference
Institute of Biomedical Informatics, National Yang Ming Chiao Tung University, Taipei, Taiwan · Institute of Health Data Science, Chang Gung University, Taoyuan, Taiwan · Department of Artificial Intelligence, Chang Gung University, Taoyuan, Taiwan +1
Does Theory of Mind Improvement Really Benefit Human-AI Interactions? Empirical Findings from Interactive Evaluations
Apr 28, 2026Nanxu Gong, Zixin Chen, Haotian Li +5LLM Evaluation Theory of Mind
Arizona State University · Hong Kong University of Science and Technology · Microsoft Research Asia +1
LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation
Apr 28, 2026Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2LLM Evaluation LLM Self-Refinement
Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA
Large language models eroding science understanding: an empirical study of malignment
Apr 28, 2026Harry Collins, Hartmut Grote, Paul Newbury +2LLM Evaluation Knowledge Conflicts in Language Models
i · ii · iii
The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive
Apr 28, 2026Alex Bogdan, Adrian de Valois-FranklinLLM Evaluation Language Model Generation Evaluation
Evolutionairy AI Toronto, Canada
Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure
Apr 28, 2026David Hartmann, Manuel Tonneau, Angelie Kraft +7LLM Evaluation LLM Auditing
Weizenbaum Institute · TU Berlin · University of Oxford +9
SciEval: A Benchmark for Automatic Evaluation of K-12 Science Instructional Materials
Apr 28, 2026Zhaohui Li, Peng He, Zhiyuan Chen +4LLM Evaluation AI in Education
Department of Computer Science and Engineering, University at Buffalo, USA · Department of Teaching and Learning, Washington State University, USA · College of Chemistry, Beijing Normal University, China +1
The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models
Apr 28, 2026Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D Khemlani +1LLM Evaluation LLM Information Extraction
1New Delhi, India · JigsawStack, Inc. 2San Francisco, CA, USA · 3Durgapur, WB, India
Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models
Apr 28, 2026Michael Rumiantsau, Ivan FokeevLLM Evaluation LLM Grounding
Cube

Apr 30, 2026, cs.AIJ/K move · Enter open · S save

Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactions

Zhuoran Pan, Yue Li, Zhi Guan, Jianbin Hu+1

School of Computer Science Peking University Beijing, PA 100871 · Taiyuan University of Technology Taiyuan, Shanxi, PA 030024 · Peking University Beijing, PA 100871

Financial Services LLM Evaluation Decentralized Finance

Open paper AskPDF arXiv

The emergence of Large Language Models (LLMs) offers a transformative interface for Web3, yet existing benchmarks fail to capture the complexity of translating high-level user intents into functionally correct, state-dependent on-chain transactions. We present \textsc{Intent2Tx}, a high-fidelity benchmark featuring 29,921 single-step and 1,575 multi-step instances meticulously derived from 300 days of real-world Ethereum mainnet traces. Unlike prior works that rely on synthetic instructions, \textsc{Intent2Tx} grounds natural language intents in real-world protocol interactions across 11 categories, including diverse long-tail Decentralized Finance (DeFi) primitives. To enable rigorous evaluation, we propose an execution-aware framework that transcends surface-level text matching by employing differential state analysis on forked mainnet environments. Our extensive evaluation of 16 state-of-the-art LLMs reveals that while scaling and retrieval-augmentation enhance logical consistency and parameter precision, current models struggle with out-of-distribution generalization and multi-step planning. Crucially, our execution-based analysis demonstrates that syntactically valid outputs often fail to achieve intended state transitions, highlighting a significant gap in current "reasoning-to-execution" capabilities. \textsc{Intent2Tx} serves as a critical foundation for developing autonomous, reliable agents in intent-centric Web3 ecosystems. Code and data: https://anonymous.4open.science/r/Intent2Tx_Bench-97FF .

Momentum

Related topics

Latest papers 1,643