cs.AINov 13, 2026

Adversarially Attacking Symbolic Vocabulary Vulnerabilities In LLM Planners

Authors: Stephen Obadinma, Xiaodan Zhu

Organizations: Department of Electrical and Computer Engineering, Ingenuity Labs Research Institute, Queen’s University

Abstract

As large language models (LLMs) are increasingly deployed as autonomous agents, their abilities in navigating automated planning tasks have become essential. However, despite their strengths, LLMs' planning ability is not robust, particularly when they are not allowed to exploit commonsense cues in the symbolic vocabulary (e.g., names of actions, predicates, and objects) used to define planning tasks. This makes LLMs unreliable compared to symbolic planners. With recent reasoning models showing improved planning performance on tasks with obfuscated symbolic vocabularies, we utilize an adversarial attack framework to reveal to what extent the crucial weaknesses of LLM agents at joint symbolic interpretation and planning remain and whether they can reason properly when presented with severe cases. As such, our main contribution is devising an LLM-based attack model, which we call Symbolic Swapper, to automatically generate obfuscated planning domains with non-standard symbolic vocabularies that systematically break strong reasoning models' ability to plan. In doing so, we reveal how much LLMs' planning abilities can be further degraded under more advanced obfuscation schemes, allowing us to ascertain their success when having to rely on their pure planning ability. We find that attacks can decrease the planning performance of base models and even robust planning frameworks by over 70%. We further analyze the factors behind how LLM planning abilities break down and find that successful obfuscation causes models to appear to significantly degrade in reasoning quality despite them showing some inherent ability to understand the domain, revealing a gap in the perception of a task domain and being able to plan under it.

Figures & tables

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. How Far Are LLMs from Symbolic Planners? An NLP-Based Perspective

    Aug 2, 2025Ma'ayan Armony, Albert Meroño-Peñuela, Gerard CanalLLM PlanningLLM Evaluation

  2. When Planning Problems Become Hard for LLMs: Separating Structural Complexity from Semantic Interpretability

    Oct 18, 2026Mayukh Debnath, Adeyinka Adedigba, Rammohan MallipeddiLLM PlanningClassical Planning

  3. SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model

    Jun 12, 2026Xiaoxin Lu, Ranran Haoran Zhang, Rui ZhangLarge Language Model-Based Robot PlanningLLM Planning