Automated Software Testing

Overview

Automated software testing aims to check whether programs meet their requirements, behave correctly across inputs and environments, and remain reliable as they change. Approaches include model-based and specification-based testing, fuzzing, property-based and metamorphic testing, mutation analysis, search-based prioritization, and AI agents that generate tests or diagnose failures. Work increasingly applies these methods to complex settings such as web applications, APIs, autonomous systems, machine-learning software, and distributed programs, where simulations, browser traces, and runtime evidence can help expose failures. A central challenge is establishing trustworthy test oracles: coverage and passing tests do not necessarily show that a test detects real faults, while generated tests and agent judgments can be brittle, incomplete, or misleading.

Latest papers

CardsList

143 papers9 papers in the last four weeks, down 50% on the four weeks before. 0.1% of all new papers.

  1. M3^3-Test: Coverage Gaps in Joint-Perturbation Testing of Visual Question Answering

    Nov 17, 2026Enhui Wan, Xiaowan WangVLM EvaluationVLM Robustness

  2. EmbedGym: Letting Coding Agents Develop and Debug Embedded Firmware Without Physical Hardware

    Nov 16, 2026Minghui Zhao, Yiming Li, Sophie Zhu +5Software Engineering AgentsCoding Agent Benchmarks

  3. ADAPT: Agentic Diff-Aware Patch Testing for Pull Requests

    Nov 11, 2026Vahid Haratian, Atakan Akar, Eray Tüzünregression testingAutomated Software Testing

  4. Bug Finding With Agentic Symbolic Execution

    Nov 11, 2026George Granberry, Wolfgang Ahrendt, Andy Gordon +1Automated Software TestingSoftware Engineering Agents

  5. How Far Must a Test Explore? Certifying Composable Read-Write Interfaces from Their Own Random Edits

    Nov 9, 2026Jiaxuan Guo, Xingyu Ren, Jingxin Yang +2Automated Software TestingSpecification-Based Testing

  6. Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations

    Nov 5, 2026Xiaobo Chen, Genliang Zhu, Ziyuan Wang +1regression testingAutomated Program Repair

  7. EMBERS: Empirically Motivated Robustness Testing of Deep Research Systems via Defect-Guided Fuzzing

    Nov 1, 2026Shuochuan Li, Zan Wang, Dong Wang +5Fuzz TestingLanguage Model Robustness

  8. Can AI Guide Testing Without Generating Tests? A Comparative Study of AI-Assisted Unit Testing

    Oct 26, 2026Zihan Fang, Ron Asoulin-Handelman, Helen Wu +2Computer Science EducationGenerative AI in Education

  9. Build or Adopt? Custom-Built vs. General-Purpose LLM Agent Harnesses for Automotive GUI Testing

    Oct 24, 2026Arsham Gholamzadeh Khoee, Ruyu Fu, Jiaqi Wei +4LLM Agent HarnessesAutomated Software Testing

  10. ProductWebBench: Evaluating Website Continuity Beyond Task Completion

    Oct 22, 2026Jiajun Wu, Jian Yang, Yaxin Du +9Software Engineering BenchmarksCode Generation Evaluation

  11. From Prose to Properties: Testing Research Claims with ARATest for Agent-Native Science

    Oct 21, 2026Yifeng He, Jiachen Liu, Hao ChenProperty-Based TestingScientific Claim Verification

  12. Who Verifies the Builder? Toward Independent Assurance for Agent-Produced Software

    Oct 16, 2026Sion Israel Sion, Alain AprilProperty-Based TestingAI Agent Reliability

  13. Stateful Testing for Deep Learning Compilers

    Oct 15, 2026Jiatao Ge, Ruixiang Qian, Yibo Hu +5Automated Software TestingDifferential Testing

  14. Robotic-supported Data Loss Detection in Android Applications

    Oct 14, 2026Davi Freitas, Breno Miranda, Juliano IyodaAutomated Software Testing

  15. SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts

    Oct 13, 2026Giovanni Pio Delvecchio, Stefano Fantazzini, Giacomo Frisoni +4Code Generation EvaluationLLM Code Generation

  16. Grounding Deep Learning Framework Testing in Data Flow

    Oct 11, 2026Jiatao Ge, Yanzhou Mu, Yinglong Zou +3Automated Software TestingDNN Testing

  17. QuSema: Detecting Silent Bugs in Quantum Libraries via Quantum-knowledge-enhanced Agents

    Oct 7, 2026Yujin Song, Kaining Zhang, Qixin Zhang +3Fault DetectionAutomated Software Testing

  18. TestGRAD: Evolving Test Suites via Failure Pattern Momentum for SWE-Agent Ensemble

    Oct 7, 2026Pengfei He, Jiayuan Zhou, Shaowei Wang +1Automated Test GenerationAutomated Software Testing

  19. Multi-Label Perceptual Bug Detection in Video Games using Deep Learning on Gameplay Footage

    Oct 6, 2026Nahian Rifaat, Felix Morosov, Loutfouz ZamanAutomated Software TestingMulti-Label Classification

  20. DreamTest: World-Model Surrogates for Search-Based Testing of Deep Reinforcement Learning Agents

    Oct 3, 2026Qinghua Xu, Guancheng Wang, Boxi Yu +2Automated Software TestingNeural Surrogate Modeling

  21. WebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution Harness

    Oct 2, 2026Yun-Yun Tsai, Yuning Mao, Shiqi Wang +2Code GenerationCode Generation Evaluation

  22. FastCI: Efficient GPU-Intensive CI for LLM Training Frameworks

    Oct 1, 2026Tianshuo Qiao, Naiqian Zheng, Xiaopeng Liu +4Automated Software Testing

  23. Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments

    Sep 29, 2026Rohith Reddy Bellibatlu, Zichong Wang, Wenbin ZhangTool-Use EvaluationAutomated Software Testing

  24. Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models

    Sep 24, 2026Nils Kiele, Zainab Saad, Zirui Wang +2Automated Software TestingMutation Testing

  25. Scenario-Driven Neuroevolution: Using Models to Guide Test Generation for Games

    Sep 23, 2026Gijs van Cuyck, Patric Feldmeier, Jan Tretmans +1Automated Software TestingAutomated Test Generation

  26. FairTest: Search-Based Fairness Testing for Multi-Agent Reinforcement Learning Systems

    Sep 23, 2026Xiaotong Wang, Xuan XieAutomated Software TestingAlgorithmic Fairness

  27. WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective

    Sep 14, 2026Chenxu Liu, Zilu Zou, Peizhong Gao +9Automated Software TestingWeb Application Generation