cs.CLJan 30, 2023

On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex

Authors: Terry Yue Zhuo, Zhuang Li, Yujin Huang, Fatemeh Shiri, Weiqing Wang, Gholamreza Haffari, Yuan-Fang Li

Organizations: CSIRO’s Data61, Australia · Monash University, Australia

Abstract

Semantic parsing is a technique aimed at constructing a structured representation of the meaning of a natural-language question. Recent advancements in few-shot language models trained on code have demonstrated superior performance in generating these representations compared to traditional unimodal language models, which are trained on downstream tasks. Despite these advancements, existing fine-tuned neural semantic parsers are susceptible to adversarial attacks on natural-language inputs. While it has been established that the robustness of smaller semantic parsers can be enhanced through adversarial training, this approach is not feasible for large language models in real-world scenarios, as it requires both substantial computational resources and expensive human annotation on in-domain semantic parsing data. This paper presents the first empirical study on the adversarial robustness of a large prompt-based language model of code, \codex. Our results demonstrate that the state-of-the-art (SOTA) code-language models are vulnerable to carefully crafted adversarial examples. To address this challenge, we propose methods for improving robustness without the need for significant amounts of labeled data or heavy computational resources.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Harnessing non-adversarial robustness in large language models

    May 28, 2026Qinghua Zhou, Ellina Aleshina, Andrey Lovyagin +6LLM Fine-TuningNeural Network Robustness

  2. Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language

    May 15, 2026Vinayshekhar Bannihatti Kumar, Disha Makhija, Manoj Ghuhan Arivazhagan +1LLM Fine-TuningCode Language Models

  3. Noisy Test-Time Reinforcement Learning for Code LLMs

    Sep 26, 2026Xikai Yang, Hieu Trung Nguyen, Dunyuan Xu +4Language Model RobustnessRL for Code Generation