LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation
Aug 19, 2025Krishna Garg, Firoz Shaik, Sambaran Bandyopadhyay +1LLM Evaluation Language Model Generation Evaluation
University of Illinois Chicago · Adobe Research
Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings
Aug 16, 2025Jenny Y. Huang, Yunyi Shen, Dennis Wei +1Pairwise Preference Learning LLM Evaluation
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · MIT-IBM Watson AI Lab · IBM Research
Rule2Text: A Framework for Generating and Evaluating Natural Language Explanations of Knowledge Graph Rules
Aug 14, 2025Nasim Shirvani-Mahdavi, Chengkai LiLLM Evaluation Explainability Evaluation
University of Texas at Arlington, Arlington, Texas 76019, USA
Post-training for Efficient Communication via Convention Formation
Aug 8, 2025Yilun Hua, Evan Wang, Yoav ArtziLLM Evaluation LLM Post-Training
Department of Computer Science and Cornell Tech Cornell University
Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models
Jul 23, 2025Tianlin Li, Yunxiang Wei, Zhiming Li +5LLM Evaluation Code Language Models
Nanyang Technological University Singapore · Zhejiang University China · Peng Cheng Laboratory China +2
IDRBench: Understanding the Capability of Large Language Models on Interdisciplinary Research
Jul 21, 2025Yuanhao Shen, Daniel Xavier de Sousa, Ricardo Marçal +2LLM Evaluation Large Language Model-Guided Scientific Discovery
Department of Electrical and Computer Engineering & Ingenuity Labs Research Institute, Queen’s University, Canada · Instituto Federal de Goiás, Anápolis, Brazil · National Research Council Canada
FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data
Jul 14, 2025Tao Feng, Haozhen Zhang, Zijie Lei +5LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · Nanyang Technological University · Meta Monetization AI +2
Detoxify: A framework for abusive text transformation using LLMs
Jul 14, 2025Rohitash Chandra, Jiyong Choi, Jayesh SonawaneLLM Evaluation
Transitional Artificial Intelligence Research Group, School of Mathematics and Statistics, UNSW, Sydney, Australia · Centre for Artificial Intelligence and Innovation, Pingla Institute, Sydney, Australia · Centre for Artificial Intelligence and Innovation, Pingla
LLM Bidders Preserve the Mechanism-Level Orderings of Human Bidders
Jul 12, 2025Anand Shah, Kehang Zhu, Yanchen Jiang +4LLM Evaluation Human Behavior Simulation
MIT & NBER
LLM-Based Social Simulations Require a Boundary
Jun 24, 2025Zengqing Wu, Run Peng, Takayuki Ito +2LLM Evaluation Human Behavior Simulation
University of Osaka · University of Michigan · Kyoto University +1
Measuring Intent Comprehension in LLMs
Jun 19, 2025Nadav Kunievsky, James A. EvansPrompt Sensitivity LLM Evaluation
Knowlesdge Lab, University of Chicago, Chicago, Illinois, USA · Knowledge Lab, Data Science Institute, Department of Sociology, University of Chicago, Chicago, Illinois, USA.
Sarc7: Evaluating Sarcasm Detection and Generation with Seven Types and Emotion-Informed Techniques
May 31, 2025Raina Gao, Alyssa Jeong, Lang Xiong +4LLM Evaluation Text Generation
Algoverse AI Research
SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
May 29, 2025Zixiang Xu, Yanbo Wang, Yue Huang +13LLM Evaluation AI Agent Benchmarks
Mohamed bin Zayed University of Artificial Intelligence · University of Southern California · University of Notre Dame +3
PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics
May 29, 2025Atharva Naik, Prakam, Yash Mathur +6Logical Reasoning LLM Evaluation
Carnegie Mellon University · Independent Researcher · Ohio State University
R3: Robust Rubric-Agnostic Reward Models
May 19, 2025David Anugraha, Zilu Tang, Lester James V. Miranda +5Reward Modeling LLM Evaluation
Department of Computer Science, Stanford University · Department of Computer Science, Boston University · Allen Institute for AI +4
Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?
May 11, 2025Weixian Waylon Li, Hyeonjun Kim, Mihai Cucuringu +1LLM Evaluation Quantitative Finance
AIAI, School of Informatics The University of Edinburgh Edinburgh, United Kingdom · Global Finance Research Center Sungkyunkwan University Seoul, Republic of Korea · Dept. of Mathematics; Dept. of Statistics & OMI University of California, Los Angeles; University of Oxford United States; United Kingdom
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer Inference LLM Evaluation
University of Edinburgh · Cohere · Meta
Evaluating LLMs on Chinese Topic Constructions: A Research Proposal Inspired by Tian et al. (2024)
Apr 21, 2025Xiaodong YangLLM Evaluation Language Model Probing
College of Foreign Languages Zhejiang University of Technology 288, Liuhe Rd., Xihu District, Hangzhou, Zhejiang, 310023 China
DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain
Apr 18, 2025Enhao Huang, Pengyu Sun, Shuxun Wang +13LLM Evaluation Software Vulnerability Detection
Zhejiang University Hangzhou, China · The State Key Laboratory of Blockchain and Data Security · DMind.ai Singapore +1
SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA
Apr 10, 2025Sher Badshah, Ali Emami, Hassan SajjadLLM Evaluation Language Model Generation Evaluation
Dalhousie University · Emory University
Scaling Evaluation-time Compute with Reasoning Models as Evaluators
Mar 25, 2025Seungone Kim, Ian Wu, Jinu Lee +8LLM Evaluation Language Model Generation Evaluation
CMU · UIUC2 · KAIST AI3 +3
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Mar 7, 2025Michael Krumdick, Charles Lovering, Varshini Reddy +2LLM Evaluation LLM-as-a-Judge
Kensho Technologies · MIT
Large Language Models for Cryptocurrency Transaction Analysis: A Bitcoin Case Study
Jan 30, 2025Yuchen Lei, Yuexin Xiang, Rafael Dowsley +5LLM Evaluation LLMs for Cybersecurity
School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China · Faculty of Information Technology, Monash University, Clayton, VIC 3800, Australia · CSIRO’s Data61, Eveleigh, NSW 2015, Australia +2
Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions
Nov 15, 2024Yutao Hou, Yajing Luo, Zhiwen Ruan +4Multi-Hop QA LLM Evaluation
Shanghai University of Finance and Economics · Southern University of Science and Technology · University of Edinburgh +1
Growing a Tail: Increasing Output Diversity in Large Language Models
Nov 5, 2024Michal Shur-Ofry, Bar Horowitz-Amsalem, Adir Rahamim +1LLM Evaluation LLM Prompting
Law Faculty, Hebrew University of Jerusalem; Jerusalem, Israel. · Faculty of Computer Science, Technion – Israel Institute of Technology; Haifa, Israel.
Evaluation of OpenAI o1: Opportunities and Challenges of AGI
Sep 27, 2024Tianyang Zhong, Zhengliang Liu, Yi Pan +74LLM Evaluation Code Generation
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
Jul 16, 2024Haodong Duan, Xinyu Fang, Junming Yang +41VLM Evaluation LLM Evaluation
CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
Apr 9, 2024Yu Zhou, Xingyu Wu, Jibin Wu +2LLM Evaluation Causal Discovery
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR 999077, China · College of Computer Science, Chongqing University, Chongqing 400044, China
UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking
Date pendingSarfraz Ahmad, Hasan Iqbal, Momina Ahsan +6LLM Evaluation Factual Consistency Evaluation
MBZUAI
Limits of LLM Text Detectors in Education
Date pendingLukas Gehring, Benjamin Paa\ssenLLM Evaluation AI-Generated Text Detection
Faculty of Technology, Bielefeld University

Aug 19, 2025, cs.CLJ/K move · Enter open · S save

Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

Krishna Garg, Firoz Shaik, Sambaran Bandyopadhyay, Cornelia Caragea

University of Illinois Chicago · Adobe Research

LLM Evaluation Language Model Generation Evaluation LLM Prompting AI-Assisted Scientific Research Generative AI Evaluation

Open paper AskPDF arXiv

As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and essential. We introduce Scientific Introduction Generation (SciIG), a task that evaluates LLMs' ability to produce coherent introductions from titles, abstracts, and related works. Curating new datasets from NAACL 2025 and ICLR 2025 papers, we assess five state-of-the-art models, including both open-source (DeepSeek-v3, Gemma-3-12B, LLaMA 4-Maverick, MistralAI Small 3.1) and closed-source GPT-4o systems, across multiple dimensions: lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality. Our comprehensive framework combines automated metrics with LLM-as-a-judge evaluations. Results demonstrate LLaMA-4 Maverick's superior performance on most metrics, particularly in semantic similarity and faithfulness. Moreover, three-shot prompting consistently outperforms fewer-shot approaches. These findings provide practical insights into developing effective research writing assistants and set realistic expectations for LLM-assisted academic writing. To foster re- producibility and future research, we publicly release all code and datasets.

Momentum

Related topics

Latest papers 1,631