Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoning

Jooyoung Kim; Wonje Choi; Younguk Song; Honguk Woo

cs.AIMar 19, 2026

Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoning

Authors: Jooyoung Kim, Wonje Choi, Younguk Song, Honguk Woo

Organizations: Department of Computer Science and Engineering, Sungkyunkwan University

Abstract

Recent advances in Vision-Language Models (VLMs) have enabled video-instructed robotic programming, allowing agents to interpret video demonstrations and generate executable control code. We formulate video-instructed robotic programming as a cross-domain adaptation problem, where perceptual and physical differences between demonstration and deployment induce procedural mismatches. However, current VLMs lack the procedural understanding needed to reformulate causal dependencies and achieve task-compatible behavior under such domain shifts. We introduce NeSyCR, a neurosymbolic counterfactual reasoning framework that enables verifiable adaptation of task procedures, providing a reliable synthesis of code policies. NeSyCR abstracts video demonstrations into symbolic trajectories that capture the underlying task procedure. Given deployment observations, it derives counterfactual states that reveal cross-domain incompatibilities. By exploring the symbolic state space with verifiable checks, NeSyCR proposes procedural revisions that restore compatibility with the demonstrated procedure. NeSyCR achieves a 31.14% improvement in task success over the strongest baseline Statler, showing robust cross-domain adaptation across both simulated and real-world manipulation tasks.

Figures & tables

Figure 1: Overview of NeSyCR in a drawer-organizing task scenario. (Left) Illustration of the domain gap between the demonstration and deployment. (Middle) Overview of NeSyCR framework, which generates an adapted procedure via neurosymbolic counterfactual reasoning. (Right) Outcome of the adapted procedure, showing that NeSyCR successfully executes the task via a grounded code policy.

Figure 2: Symbolic world model construction

Figure 3: Neurosymbolic counterfactual adaptation

Panel of Figure 4 — Figure 4: Analysis on (a) domain gap and (b) task complexity gap

Figure 5: Visualization of a cross-domain demo-to-code task

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Example scene of Obstruction and Object affordance

Task	Level 1	Level 2
Pick & Place	A cube block stacked on preceding block.	Multiple cube blocks stacked on preceding block.
Sweep	A chess piece starts in the wrong box.	Multiple chess pieces start in the wrong box.
Rotate	Hinge lid starts closed.	Objects start stacked on top of the closed lid.
Slide	Target drawer starts closed.	Target drawer starts closed, other obstructs from above.

Appendix

Table 4: Description of task variations across obstruction levels

Figure 7: Example scene of Kinematic configuration and Gripper type

Figure 8: Example scenes for the subtasks

Figure 9: Example scenes for each task complexity level

Domain Factors \Task Complexity	Low	Medium	High	Total
Obstruction & Object Affordance	60	60	40	160
Kinematic Config & Gripper	60	60	40	160
Combination	40	40	40	120
Total Scenarios	160	160	120	440

Appendix

Table 5: Experimental scenarios for model evaluation

Panel of Table 6 — Table 6: Perception API primitives

Panel of Table 7 — Table 7: Action API primitives

Figure 10: Example scenes from the real-world environment used in our experiments

Panel of Figure 11 — Figure 11: Predicate set used for symbolic state representation

Panel of Figure 12 — Figure 12: NeSyCR symbolic world model construction high-level flow

Panel of Figure 13 — Figure 13: Example of symbolic state and action operator extracted from the demonstration

Panel of Figure 14 — Figure 14: NeSyCR neurosymbolic counterfactual adaptation high-level flow

Panel of Figure 15 — Figure 15: Example of counterfactual identification and exploration for the demonstration

Hyperparameters	Value
Max explorations	10 (low/medium), 20 (high)
	(Obstruction/Kinematic)
	15 (low/medium), 30 (high)
	(Combination)

Appendix

Table 8: NeSyCR hyperparameters

Panel of Figure 16 — Figure 16: Example of task specification

Baseline	Natively Code	Explicit Adapt	Replanning	World Model	Symbolic Tool
Demo2Code	✓	✗	✗	✗	✗
GPT4V-Robotics	✓	✓	✓	✗	✗
Critic-V	✗	✓	✗	✗	✗
MoReVQA	✗	✓	✗	✗	✗
Statler	✗	✓	✓	✓	✗
LLM-DM	✗	✓	✓	✓	✓

Appendix

Table 9: Comparison of baseline methods

Panel of Figure 17 — Figure 17: Demo2Code high-level flow

Panel of Figure 18 — Figure 18: GPT4V-Robotics high-level flow

Panel of Figure 19 — Figure 19: Critic-V high-level flow

Hyperparameters	Value
Max critique refinement	10 (low/medium), 20 (high)
	(Obstruction/Kinematic)
	15 (low/medium), 30 (high)
	(Combination)

Appendix

Table 10: Critic-V hyperparameters

Panel of Figure 20 — Figure 20: MoReVQA high-level flow

Panel of Figure 21 — Figure 21: Statler high-level flow

Hyperparameters	Value
Max planning iteration	10 (low/medium), 20 (high)
	(Obstruction/Kinematic)
	15 (low/medium), 30 (high)
	(Combination)

Appendix

Table 11: Statler hyperparameters

Panel of Figure 22 — Figure 22: LLM-DM high-level flow

Hyperparameters	Value
Max domain refinement	10 (low/medium), 20 (high)
	(Obstruction/Kinematic)
	15 (low/medium), 30 (high)
	(Combination)

Appendix

Table 12: LLM-DM hyperparameters

Figure 23: An expanded visualization of the adaptation process illustrated in Figure 1

Figure 24: An expanded visualization of the task execution illustrated in Figure 1

Figure 25: Visualization of real-world experiment in Table 2

Robot Learning

Robot learning develops methods for robots to acquire skills and adapt from demonstrations, interaction, data, and physical models.

See all 321 papers

One-shot Human Video-Guided Adaptation for Vision-Language-Action Models
Oct 14, 2026Qi Chen, Yuwei Wu, Quan Chen +5VLA Model Adaptation Vision-Language-Action Models
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science and Technology, Beijing Institute of Technology. · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University. · AInnovation Technology Group Co., Ltd.
Build on Priors: Vision--Language--Guided Neuro-Symbolic Imitation Learning for Data-Efficient Real-World Robot Manipulation
Apr 4, 2026Pierrick Lorang, Johannes Huemer, Timothy Duggan +3Robot Imitation Learning Robotic Manipulation
Department of Computer Science, Tufts University, Medford, MA, USA · Complex Dynamical Systems, Austrian Institute of Technology GmbH (AIT), Vienna, Austria
SeeTraceAct: Visibility-Aware Latent Planning from Cross-Embodiment Demonstration Videos
Jun 1, 2026Jaehyeon Son, Junhyun Kim, Kyle Kam +7Robot Skill Learning Generalization in Robotic Manipulation
Georgia Institute of Technology · Allen Institute for AI · Johns Hopkins University +1

Oct 14, 2026J/K move · Enter open · S save

One-shot Human Video-Guided Adaptation for Vision-Language-Action Models

Qi Chen, Yuwei Wu, Quan Chen, Kunpeng Gao+4

Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science and Technology, Beijing Institute of Technology. · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University. · AInnovation Technology Group Co., Ltd.

VLA Model Adaptation Vision-Language-Action Models Test-Time Adaptation of VLA Models Robotic Manipulation Video Representation Learning

Open paper AskPDF arXiv

Vision-language-action (VLA) models have recently achieved remarkable progress by leveraging large-scale robot datasets and foundation models for general-purpose robot manipulation. However, their ability to rapidly adapt to novel manipulation tasks beyond the training distribution remains limited. Existing approaches often require task-specific robot demonstrations and additional training, which are labor- and compute-intensive, limiting the scalability of VLA adaptation to open-world scenarios. To reduce this dependence on target-task robot data, we propose CapPro, a framework that enables a VLA policy to adapt to previously unseen manipulation tasks using only a single human demonstration video. CapPro consists of two components. First, a phase-aware human video representation module constructs a compact video representation through learned visual token selection and learns task-relevant query representations under phase-caption supervision. Second, a video-conditioned adaptation module jointly incorporates the compact video and query-derived representations into the action-conditioning context of the pretrained VLA policy. % This design introduces task-specific guidance while largely preserving the original VLA action-generation backbone. In this way, the human demonstration provides task-specific execution guidance rather than an embodiment-aligned trajectory to imitate, without requiring target-task robot demonstrations or target-task fine-tuning. Experiments across source and unseen target tasks show that CapPro improves average performance across unseen target groups while maintaining comparable performance on the source tasks. These results demonstrate the effectiveness of human-video guidance for cross-task VLA adaptation.

Abstract

Robot Learning

Explore similar work

One-shot Human Video-Guided Adaptation for Vision-Language-Action Models

Build on Priors: Vision--Language--Guided Neuro-Symbolic Imitation Learning for Data-Efficient Real-World Robot Manipulation

SeeTraceAct: Visibility-Aware Latent Planning from Cross-Embodiment Demonstration Videos