cs.CRApr 15, 2026

Challenges and Future Directions in Agentic Reverse Engineering Systems

Authors: Salem Radey, Jack West, Kassem Fawaz

Organizations: University of Wisconsin-Madison, Madison, USA

Abstract

Agentic systems built on large language models (LLMs) are increasingly being used for complex security tasks, including binary reverse engineering (RE). Despite recent growth in popularity and capability, these systems continue to face limitations in realistic settings. Cutting-edge systems still fail in complex RE scenarios that involve obfuscation, timing, and unique architecture. In this work, we examine how agentic systems perform reverse engineering tasks with static, dynamic, and hybrid agents. Through an analysis of existing agentic tool usage, we identify several limitations, including token constraints, struggles with obfuscation, and a lack of program guardrails. From these findings, we outline current challenges and position future directions for system designers to overcome from a security perspective.

Figures & tables

Explore similar work

CardsList
  1. The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark

    Aug 11, 2026Jeremy Spence, Nicholas Assaderaghi, Feng Xiao +9Software Reverse EngineeringAI Agent Security Benchmarks

  2. Automatically Attacking Software Reverse Engineering AI Agents

    May 28, 2026Brian Crawford, Justin Phillips, Patrick McClureSoftware EngineeringAdversarial Prompt Generation

  3. Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing

    Aug 16, 2026Israt Moyeen Noumi, Tarannum Ahmed Nowshin, Md. Mehedi Hasan Nipu +3LLM Agent SecurityAutomated Penetration Testing