Critic Model

Critic models are auxiliary components in machine learning systems designed to evaluate the quality of outputs generated by other models, such as large language models (LLMs) or reinforcement learning agents. Current research focuses on improving the accuracy and efficiency of these critics, often employing techniques like actor-critic architectures, Monte Carlo Tree Search, and various loss functions tailored to specific tasks (e.g., image classification, code evaluation, mathematical reasoning). The development of effective critic models is crucial for enhancing the reliability and performance of LLMs and reinforcement learning agents, leading to improvements in diverse applications ranging from question answering and code generation to robotics and game playing.

Papers

November 10, 2024

CriticAL: Critic Automation with Language Models
Michael Y. Li, Vivek Vajipey, Noah D. Goodman, Emily B. Fox
Language Model Critic Model Model Criticism Scientific Model

October 29, 2024

Human-Readable Programs as Actors of Reinforcement Learning Agents Using Critic-Moderated Evolution
Senne Deproost, Denis Steckelmacher, Ann Nowé
Deep Reinforcement Learning Black Box Reinforcement Learning Agent Critic Model Actor Loss Programmatic Reinforcement Learning RL Algorithm

October 12, 2024

TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning
Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, Gerhard Neumann
Policy Reinforcement Learning Critic Model Trajectory Generator

October 2, 2024

Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks
Xingxuan Li, Weiwen Xu, Ruochen Zhao, Fangkai Jiao, Shafiq Joty, Lidong Bing
Large Language Model Reinforcement Learning Complex Reasoning Task Planning Retrieval Augmentation Complex Task Critic Model

September 23, 2024

Critic Loss for Image Classification
Brendan Hogan Rappazzo, Aaron Ferber, Carla Gomes
Image Classification Cross Entropy Loss Multiple Critic Critic Model

September 15, 2024

Critic as Lyapunov function (CALF): a model-free, stability-ensuring agent
Pavel Osinenko, Grigory Yaremenko, Roman Zashchitin, Anton Bolychev, Sinan Ibrahim, Dmitrii Dobriborsci
Reinforcement Learning Model Predictive Control Lyapunov Function Multiple Critic Critic Model Safe Agent

August 29, 2024

Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Critic
Xin Zheng, Jie Lou, Boxi Cao, Xueru Wen, Yuqiu Ji, Hongyu Lin, Yaojie Lu, Xianpei Han, Debing Zhang, Le Sun
Large Language Model Chain of Thought Reasoning Ability Critique Ability Reasoning Performance Multiple Critic Critic Model

August 28, 2024

SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models
Dian Yu, Baolin Peng, Ye Tian, Linfeng Song, Haitao Mi, Dong Yu
Large Language Model Critic Model

August 13, 2024

Heavy-Ball Momentum Accelerated Actor-Critic With Function Approximation
Yanjie Dong, Haijun Zhang, Gang Wang, Shisheng Cui, Xiping Hu
Markov Decision Process Function Approximation Reinforcement Learning Task Critic Model Heavy Ball Stochastic Policy Gradient

June 28, 2024

LLM Critics Help Catch LLM Bugs
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, Jan Leike
Human Feedback LLM Fine Tuning LLM Reasoning LLM Evaluation Critic Model Model Criticism

June 11, 2024

Merging Improves Self-Critique Against Jailbreak Attacks
Victor Gallego
Large Language Model Jailbreak Attack Adversarial Prompt Self Feedback Critic Model Adversarial Manipulation

May 3, 2024

Closing the Gap: Achieving Global Convergence (Last Iterate) of Actor-Critic under Markovian Sampling with Neural Network Parametrization
Mudit Gaur, Amrit Singh Bedi, Di Wang, Vaneet Aggarwal
Neural Network Global Convergence State Action Space Sample Complexity Bound Critic Model Markovian Sampling Local Optimality Gradient Dominance Property

April 18, 2024

Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, Dong Yu
Large Language Model Language Model Complex Reasoning Search Query Self Improvement Critic Model

March 25, 2024

Weak Convergence Analysis of Online Neural Actor-Critic Algorithms
Samuel Chun-Hei Lam, Justin Sirignano, Ziheng Wang
Neural Network Policy Gradient Actor Critic Algorithm Critic Model Weak Convergence Online Deep Learning

March 8, 2024

Can Large Language Models Play Games? A Case Study of A Self-Play Approach
Hongyi Guo, Zhihan Liu, Yufeng Zhang, Zhaoran Wang
Case Study Monte Carlo Tree Search Video Game Self Play Iterative Pruning Critic Model

March 1, 2024

A Case for Validation Buffer in Pessimistic Actor-Critic
Michal Nauman, Mateusz Ostaszewski, Marek Cygan
Case Relevance Critic Model Bentonite Buffer Time Varying Objective Optimistic Learning

February 19, 2024

A Critical Evaluation of AI Feedback for Aligning Large Language Models
Archit Sharma, Sedrick Keh, Eric Mitchell, Chelsea Finn, Kushal Arora, Thomas Kollar
Reinforcement Learning Critical Review AI Feedback Critic Model

January 14, 2024

Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, Lei Meng
Language Model Reinforcement Learning Text Generation Sparse Reward Intrinsic Reward Critic Model

December 21, 2023

Critic-Guided Decision Transformer for Offline Reinforcement Learning
Yuanfu Wang, Chao Yang, Ying Wen, Yu Liu, Yu Qiao
Offline Reinforcement Learning Value Function Decision Transformer Critic Model Return Conditioned Supervised Learning

October 7, 2023

Critique Ability of Large Language Models
Liangchen Luo, Zi Lin, Yinxiao Liu, Lei Shu, Yun Zhu, Jingbo Shang, Lei Meng
Critique Ability Critic Model Realistic Evaluation Lower Critique Accuracy