Interface-Aware Trajectory Reconstruction of Limited Demonstrations for Robot Learning

Demiana R. Barsoum; Mahdieh Nejati Javaremi; Larisa Y. C. Loke; Brenna D. Argall

cs.ROFeb 26, 2026

Interface-Aware Trajectory Reconstruction of Limited Demonstrations for Robot Learning

Authors: Demiana R. Barsoum, Mahdieh Nejati Javaremi, Larisa Y. C. Loke, Brenna D. Argall

Organizations: Northwestern University & Shirley Ryan AbilityLab, Chicago, Illinois, USA

Abstract

Assistive robots offer agency to humans with severe motor impairments. Often, these users control high-DoF robots through low-dimensional interfaces, such as using a 1-D sip-and-puff interface to operate a 6-DoF robotic arm. This mismatch results in having access to only a subset of control dimensions at a given time, imposing unintended and artificial constraints on robot motion. As a result, interface-limited demonstrations embed suboptimal motions that reflect interface restrictions rather than user intent. To address this, we present a trajectory reconstruction algorithm that reasons about task, environment, and interface constraints to lift demonstrations into the robot's full control space. We evaluate our approach using real-world demonstrations of ADL-inspired tasks performed via a 2-D joystick and 1-D sip-and-puff control interface, teleoperating two distinct 7-DoF robotic arms. Analyses of the reconstructed demonstrations and derived control policies show that lifted trajectories are faster and more efficient than their interface-constrained counterparts while respecting user preferences.

Figures & tables

Figure 1. Limited interfaces and control modes. Sip/Puff Breeze ™ with Headset ( left ). Image © Origin Instruments Origin Instruments, 2025 , reproduced with permission. 2-D joystick ( right ). An example of modal partitioning of the robot control space ( bottom , end-effector translation $\vec{v}$ and rotation $\vec{\omega}$ velocity vectors, and gripper $g$ open/close.) The left image showcases the two control interfaces used to collect demonstrations: a sip/puff and a joystick. The left is a table depicting the paradigms for each of the control interfaces. the sip/puff requires 7 modes to control each dimension plus the gripper control while the 2-D joystick offers four modes that give access to 2 dimensions at a time.

Figure 2. Illustrative example comparing raw, smoothed, and reconstructed trajectories from two different interfaces. Left: The demonstrator intentionally issues lower-dimensional (1-D) commands through a higher-dimensional (2-D) interface. Right: The demonstration instead is constrained by a lower-dimensional (1-D) interface. Illustrative example of comparing raw, smoothed, and reconstructed trajectories. The figure on the right depicts an example when he demonstrator intentionally uses a higher-dimensional interface in a 1-D manner. Thus, the raw trajectory is a motion in the left, followed by motion up. The reconstructed trajectory matches the raw trajectory as this respects the user's intention and preference to use the interface in a 1-D manner. The figure on the right depicts an example of a 1-D interface (same right to up motion). However, the reconstructed trajectory couples the right and up motion since the user's preference cannot be embedded due to the nature of the interface. In both figures, the smoothed trajectory softens the trajectory, removing the user's preference/intention in the left figure.

Figure 3. Experimental setup of each of the tasks. The xArm7 robotic arm with a pincher gripper is pictured in the same initial condition for each task. Image of the 7-DoF robotic arm used for the experiments from UFactory, xArm7. The robot is white and includes a pincher gripper. The robot is in its initial configuration we name ``home". There are six number of tasks: Pick-and-place, Pour, Peg-in-hole, Stir, Brush Hair, and Cut Food.

Figure 4. Percentage of dimensions activated across the full trajectory, averaged over all demonstrations. Raw (top row) and reconstructed (bottom row) trajectories collected with the joystick (left column) and sip/puff (right column. This figure (for the xArm7 robotic arm) includes four confusion matrices (two-by-two). The first column of matrices is for the joystick control interface while the second column of matrices is for the sip/puff control interface. The top two matrices show the percentage of dimensions activated during the raw demonstrations while the bottom two matrices show the percentage of dimensions activated after our reconstruction (lifting) algorithm has been applied. Each row in the matrix represents one of the tasks. The joystick raw demonstrations (top left) go up to 2-D, while our reconstructed joystick demonstrations (bottom left) goes up to 6-D. Similarly, the sip/puff raw demonstrations (top right) are only one-dimensional, while with our method (bottom right), demonstrations are lifted to 4-D.

Figure 5. Evolution of dimension activations using the 2-D joystick for the Pick-and-place task. Activations in the raw demonstration ( top ) are 1-D or 2-D only, while the reconstructed demonstration ( bottom ) lifts as high as the full 6-D. This figure is a timeseries plot depicting the evolution of dimension activation for a demonstration collection using the 2-D joystick joystick for the Pick-and-place task. The raw demonstration (top) shows 1-D to 2-D activation while the reconstructed demonstration (bottom) depicts the lifting into higher dimensions. In this case, up to the full 6-D. Environment and task constraints are demarcated in the timeseries.

Interface	Pick-and-Place	Pour (1)	Peg-in-hole	Stir	Brush Hair	Cut Food (3)
Joystick	4.4 $\pm$ 0.8	4.8 $\pm$ 1.0	4.0 $\pm$ 0.0	4.0 $\pm$ 0.0	4.0 $\pm$ 0.0	4.0 $\pm$ 0.0
Sip/Puff	3.2 $\pm$ 0.7	4.0 $\pm$ 0.0	2.5 $\pm$ 0.5	2 $\pm$ 0.0	2.7 $\pm$ 0.9	3.2 $\pm$ 0.7

Table 1. Maximum number of controllable dimensions for a composed segment, averaged over all demonstrations.

Figure 6. Comparison between interface-aware reconstruction method (magenta) and smoothing filter (green) on the Peg-in-hole raw trajectory (black, left ) and the corresponding demonstration on the 7-DoF xArm robotic arm ( right ). Numbers indicate task progression. This figure shows the 3-D trajectory of a Peg-in-Hole demonstration (left) and the corresponding demonstration on the xArm robotic arm (right).

Dataset	Metric	Pick-and-place	Pour	Peg-in-hole	Stir	Brush Hair	Cut Food	Average
Raw	Time (s)	46.7 $\pm$ 3.5	50.1 $\pm$ 9.0	11.3 $\pm$ 0.6	46.4 $\pm$ 6.6	17.7 $\pm$ 1.8	21.3 $\pm$ 4.2	–
Raw	Dist. (m)	2.1 $\pm$ 0.3	2.1 $\pm$ 0.3	0.6 $\pm$ 0.0	2.5 $\pm$ 0.5	0.8 $\pm$ 0.0	1.0 $\pm$ 0.2	–
Smoothed	Time (s)	46.7 $\pm$ 3.5 (0.0%)	50.1 $\pm$ 9.0 (0.0%)	11.3 $\pm$ 0.6 (0.0%)	46.4 $\pm$ 6.6 (0.0%)	17.7 $\pm$ 1.8 (0.0%)	21.3 $\pm$ 4.2 (0.0%)	0.0%
Smoothed	Dist. (m)	2.1 $\pm$ 0.3 (-0.4%)	2.1 $\pm$ 0.3 (-0.4%)	0.6 $\pm$ 0.0 (-0.3%)	2.5 $\pm$ 0.5 (-0.4%)	0.8 $\pm$ 0.0 (-0.3%)	1.0 $\pm$ 0.2 (-1.1%)	-0.5%
Recon.	Time (s)	30.5 $\pm$ 2.3 (-35.0%)	38.3 $\pm$ 7.3 (-23.6%)	9.3 $\pm$ 0.4 (-17.4%)	36.7 $\pm$ 6.9 (-21.0%)	13.5 $\pm$ 1.7 (-23.9%)	13.9 $\pm$ 3.6 (-34.6%)	-25.9%
Recon.	Dist. (m)	1.7 $\pm$ 0.2 (-16.1%)	1.9 $\pm$ 0.2 (-9.0%)	0.5 $\pm$ 0.0 (-13.7%)	2.3 $\pm$ 0.5 (-7.6%)	0.8 $\pm$ 0.0 (-9.1%)	0.9 $\pm$ 0.2 (-13.2%)	-11.5%

Table 2. Joystick interface execution time and path length, averaged over demonstrations, by and across tasks, on the xArm7. Percentages reflect changes from raw demonstrations.

Dataset	Metric	Pick-and-place	Pour (1)	Peg-in-hole	Stir	Brush Hair	Cut Food (3)	Average
Raw	Time (s)	44.5 $\pm$ 14.0	37.9	16.8 $\pm$ 0.3	24.5 $\pm$ 1.7	16.3 $\pm$ 1.8	18.3 $\pm$ 2.7	–
Raw	Dist. (m)	1.8 $\pm$ 0.6	2.5	0.8 $\pm$ 0.1	1.5 $\pm$ 0.1	0.9 $\pm$ 0.1	1.2 $\pm$ 0.1	–
Smoothed	Time (s)	44.5 $\pm$ 14.0 (0.0%)	37.9 (0.0%)	16.8 $\pm$ 0.3 (0.0%)	24.5 $\pm$ 1.7 (0.0%)	16.3 $\pm$ 1.8 (0.0%)	18.3 $\pm$ 2.7 (0.0%)	0.0%
Smoothed	Dist. (m)	1.8 $\pm$ 0.6 (-2.1%)	2.4 (-3.1%)	0.8 $\pm$ 0.0 (-2.5%)	1.5 $\pm$ 0.1 (-2.6%)	0.9 $\pm$ 0.1 (-4.9%)	1.1 $\pm$ 0.1 (-3.3%)	-3.1%
Reconstructed	Time (s)	29.7 $\pm$ 9.0 (-33.2%)	28.2 (-25.7%)	10.4 $\pm$ 1.0 (-38.3%)	15.9 $\pm$ 1.0 (-35.3%)	11.7 $\pm$ 1.8 (-28.6%)	12.3 $\pm$ 2.1 (-33.1%)	-33.4%
Reconstructed	Dist. (m)	1.5 $\pm$ 0.6 (-18.4%)	2.3 (-8.3%)	0.4 $\pm$ 0.1 (-49.9%)	1.3 $\pm$ 0.1 (-17.0%)	0.8 $\pm$ 0.0 (-18.9%)	1.0 $\pm$ 0.2 (-12.0%)	-23.6%

Table 3. Sip/puff interface execution time and path length, averaged over demonstrations, by task and across tasks, on the xArm7. Percentages reflect changes from raw demonstrations. Gray-shading denotes fewer than 5 collected demonstrations, with the number of successful demonstrations collected noted in parenthesis.

Task	Raw		Reconstructed
	Joy	Sip/puff	Joy	Sip/puff
Pick-and-Place	0.002	0.001	0.007	0.004
Pour	0.002		0.026
Peg-in-hole	0.052	3.776	0.043	0.661
Stir	0.004	9e-5	0.002	2e-4
Brush Hair	0.006	0.002	0.002	0.010

Table 4. Mean Squared Error (MSE) quantification of errors between the demonstrations and learned trajectories. Successful policy rollouts are highlighted (in green). (Gray shading indicates fewer than 5 collected demonstrations. Values are unit-scaled from 0 to 1, with translation in mm and orientation in radians.)

Figure 7. Peg-in-hole task setup for the Gen2 Jaco. The image depicts the experimental setup for the Peg-in-hole task. The Kinova Gen2 Jaco 7-DoF robotic arm with a Sharpie highlighter in hand in its initial condition.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Filter Type	Hyperparameters
Low-pass Butterworth	Order: 4, Cutoff: 2
Savitzky-Golay	Window length: 11, Poly order: 3
B-spline	Degree: 3

Appendix

Table 5. Filtering methods and their corresponding hyperparameters.

Figure 8. Percentage of dimensions activated across the full trajectory, averaged over all demonstrations. Joystick raw ( top left ); joystick smoothed ( bottom left ); sip/puff raw ( top right ); sip/puff smoothed ( bottom right ). This figure includes four confusion matrices (two-by-two). The first column of matrices is for the joystick control interface while the second column of matrices is for the sip/puff control interface. The top two matrices show the percentage of dimensions activated during the raw demonstrations while the bottom two matrices show the percentage of dimensions activated after smoothing has been applied. Each row in the matrix represents one of the tasks. The joystick raw demonstrations (top left) go up to 2-D, while our reconstructed joystick demonstrations (bottom left) stays the same at 2-D. Similarly, the sip/puff raw demonstrations (top right) are only one-dimensional and same with the smoothing method (bottom right), demonstrations are up to 1-D.

Hyperparameter	Value
Number of Hidden Layers	2
Hidden Layer Sizes	[128, 64]
Activation Function	ReLU
Output Activation	Linear
Loss Function	MSE
Optimizer	Adam

Appendix

Table 6. Hyperparameters for the Multi-Layer Perceptron (MLP) Model

Sub-goal	Raw		Reconstructed
	Joystick	Sip/Puff	Joystick	Sip/Puff
Overall success?
Collision free?
Within joint limits?
Cup picked up?
Cup flipped?

Appendix

Table 7. xArm7: Sub-goal success checks for Raw and Reconstructed policies on the Pick-and-Place task .

Sub-goal	Raw		Reconstructed
	Joystick	Sip/Puff	Joystick	Sip/Puff
Overall success?
Collision free?
Within joint limits?
Cereal picked up?
Cereal in bowl?			—	—

Appendix

Table 8. xArm7: Sub-goal success checks for Raw and Reconstructed policies on the Pour Task .

Sub-goal	Raw		Reconstructed
	Joystick	Sip/Puff	Joystick	Sip/Puff
Overall success?
Collision free?
Within joint limits?
Peg inserted in hole?

Appendix

Table 9. xArm7: Sub-goal success checks for Raw and Reconstructed policies on the Peg-in-hole Task .

Sub-goal	Raw		Reconstructed
	Joystick	Sip/Puff	Joystick	Sip/Puff
Overall success?
Collision free?
Within joint limits?
Robot whisked?

Appendix

Table 10. xArm7: Sub-goal success checks for Raw and Reconstructed policies on the Stir Task .

Sub-goal	Raw		Reconstructed
	Joystick	Sip/Puff	Joystick	Sip/Puff
Overall success?
Collision free?
Within joint limits?
Brush contacted head?
Hair brushed?

Appendix

Table 11. xArm7: Sub-goal success checks for Raw and Reconstructed policies on the Brush Hair Task .

Sub-goal	Raw		Reconstructed
	Joystick	Sip/Puff	Joystick	Sip/Puff
Overall success?
Collision free?
Within joint limits?
Food cut?

Appendix

Table 12. xArm7: Sub-goal success checks for Raw and Reconstructed policies on the Cut Food Task .

Robot	Joint 1	Joint 2	Joint 3	Joint 4	Joint 5	Joint 6	Joint 7
xArm7	$\pm$ 360°	-117°, 116°	$\pm$ 360°	-6°, 225°	$\pm$ 360°	-97°, 180°	$\pm$ 360°
Gen2 Jaco	$\pm 27.7$ turns

Appendix

Table 13. Gen 2 Jaco: Joint limit specifications for Gen2 Jaco and xArm7 Ufactory, 2025 ; Kinova, 2025 .

Sub-goal	Raw		Reconstructed
	Joystick	Sip/Puff	Joystick	Sip/Puff
Overall success?
Collision free?
Cup picked up?
Cup flipped?
Cup placed down?

Appendix

Table 14. Gen 2 Jaco: Sub-goal success checks for Raw and Reconstructed policies on the Pick-and-Place task .

Sub-goal	Raw		Reconstructed
	Joystick	Sip/Puff	Joystick	Sip/Puff
Overall success?
Collision free?
Whisk picked up?
Robot whisked?
Whisk returned?

Appendix

Table 15. Gen 2 Jaco: Sub-goal success checks for Raw and Reconstructed policies on the Stir task . This task includes additional sub-goals.

Sub-goal	Raw		Reconstructed
	Joystick	Sip/Puff	Joystick	Sip/Puff
Overall success?
Collision free?
Peg inserted?

Appendix

Table 16. Gen 2 Jaco: Sub-goal success checks for Raw and Reconstructed policies on the Peg-in-hole task .

Explore similar work

See, Learn, Assist: Safe and Self-Paced Robotic Rehabilitation via Video-Based Learning from Demonstration
Mar 15, 2026Ali Alabbas, Camillo Murgia, Joanne Regan +1Assistive Robotics Robotic Control
Faculty of Mechanical Engineering, Atlantic Technological University, Galway, Ireland. · Department of Health and Nutritional Science, Atlantic Technological University, Sligo, Ireland.
Robot Learning from Human Demonstrations: Handwritten Alphabet Trajectories and Human-Likeness Evaluation
Aug 6, 2026Alperen Kenan, Paul Bremner, Manuel GiulianiRobot Skill Learning Gaussian Mixture Models
Bristol Robotics Laboratory, University of the West of England, Bristol, United Kingdom · Faculty of Electrical Engineering, Kempten University of Applied Sciences, Kempten, Germany
Touch2Robot: Robot Touch in the Human Demonstration Loop
Sep 21, 2026Shengcheng Luo, Xiaoyang Cheng, Hong Ying +6Robotic Data Collection Motion Retargeting
ShanghaiTech University · Beijing Institute for General Artificial Intelligence (BIGAI) · Shanghai Jiao Tong University +1

Mar 15, 2026J/K move · Enter open · S save

See, Learn, Assist: Safe and Self-Paced Robotic Rehabilitation via Video-Based Learning from Demonstration

Ali Alabbas, Camillo Murgia, Joanne Regan, Philip Long

Faculty of Mechanical Engineering, Atlantic Technological University, Galway, Ireland. · Department of Health and Nutritional Science, Atlantic Technological University, Sligo, Ireland.

Assistive Robotics Robotic Control Robot Motion Generation Robot Imitation Learning

Open paper AskPDF arXiv

In this paper, we propose a novel framework that allows therapists to teach robot-assisted rehabilitation exercises remotely via RGB-D video. Our system encodes demonstrations as 6-DoF body-centric trajectories using Cartesian Dynamic Movement Primitives (DMPs), ensuring accurate posture-independent spatial generalisation across diverse patient anatomies. Crucially, we execute these trajectories through a decoupled hybrid control architecture that constructs a spatially compliant virtual tunnel, paired with an effort-based temporal dilation mechanism. This architecture is applied to three distinct rehabilitation modalities: Passive, Active-Assisted, and Active-Resistive, by dynamically linking the exercise's execution phase to the patient's tangential force contribution. To guarantee safety, a Gaussian Mixture Regression (GMR) model is learned on-the-fly from the patient's own limb. This allows the detection of abnormal interaction forces and, if necessary, reverses the trajectory to prevent injury. Experimental validation demonstrates the system's precision, achieving an average trajectory reproduction error of 3.7cm and a range of motion (ROM) error of 5.5 degrees. Furthermore, dynamic interaction trials confirm that the controller successfully enforces effort-based progression while maintaining strict spatial path adherence against human disturbances.

Abstract

Explore similar work

See, Learn, Assist: Safe and Self-Paced Robotic Rehabilitation via Video-Based Learning from Demonstration

Robot Learning from Human Demonstrations: Handwritten Alphabet Trajectories and Human-Likeness Evaluation

Touch2Robot: Robot Touch in the Human Demonstration Loop