Safety Critic

Safety critics are components of artificial intelligence systems, particularly in reinforcement learning, designed to evaluate the safety of an agent's actions and prevent unsafe behaviors. Current research focuses on developing effective safety critics using various methods, including gradient-based optimization, binary Bellman operators, and latent variable models within actor-critic frameworks. These advancements aim to improve the reliability and safety of AI agents in complex environments, addressing critical challenges in deploying AI in real-world applications such as robotics and autonomous systems.

Papers

June 23, 2024

INDICT: Code Generation with Internal Dialogues of Critiques for Both Security and Helpfulness
Hung Le, Yingbo Zhou, Caiming Xiong, Silvio Savarese, Doyen Sahoo
Code Generation Natural Language Instruction Security Related Lower Critique Accuracy Code Interpreter Crime Story Safety Critic Multi Attribute Helpfulness Dataset

May 24, 2024

Counterexample-Guided Repair of Reinforcement Learning Systems Using Safety Critics
David Boetius, Stefan Leue
Constrained Optimization Third Position Repair Reinforcement Learning System Counterexample Guided Repair Safety Critic

January 23, 2024

Learning safety critics via a non-contractive binary bellman operator
Agustin Castellano, Hancheng Min, Juan Andrés Bazerque, Enrique Mallada
Reinforcement Learning Bellman Equation Bellman Operator Safety Critic

June 24, 2023

Safe Reinforcement Learning with Dead-Ends Avoidance and Recovery
Xiao Zhang, Hai Zhang, Hongtu Zhou, Chang Huang, Di Zhang, Chen Ye, Junqiao Zhao
Reinforcement Learning Safe Reinforcement Learning Prompt Recovery Safe Exploration Continuous Control Task Safety Critic Dead End

October 2, 2022

Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representation
Yannick Hogewind, Thiago D. Simao, Tal Kachman, Nils Jansen
Safe Reinforcement Learning Tetromino Pixel Observable Markov Decision Process Partial Observability Reward Maximization Safety Critic

June 26, 2022

Learning to Rearrange with Physics-Inspired Risk Awareness
Meng Song, Yuhan Liu, Zhengqin Li, Manmohan Chandraker
Reinforcement Learning LeArning Abstract Robotic Agent Risk Awareness Entropic Risk Safety Critic Physic Based Reward