A human-in-the-loop Control Paradigm for Construction Robotics: From VR-Guided Teleoperation to Semi-Autonomous Dual-Arm Operation
Organizations: Division of Integrative Systems and Design, Hong Kong University of Science and Technology, Hong Kong · Hong Kong University of Science and Technology, Hong Kong, China
Abstract
Construction tasks in hazardous and unstructured environments require both human adaptability and robotic precision, yet existing approaches often treat teleoperation and autonomy as separate control modes. This paper presents a human-in-the-loop control paradigm that enables structured transitions among VR-guided teleoperation, semi-autonomous shared control, and rule-based autonomous execution for contact-rich dual-arm construction tasks. The proposed system combines immersive VR guidance, a manually defined library of reusable skill primitives, and a shared-autonomy framework with explicit human-robot role allocation and command-priority rules. This integration allows operators to retain supervisory and corrective authority while delegating repetitive and precision-critical actions to the robot. The paradigm is validated on a dual-arm platform through representative facade-installation tasks. Results show that autonomous execution provides the best performance under well-calibrated nominal conditions, whereas semi-autonomous control improves resilience when environmental disturbances exceed the predefined autonomous pipeline's correction capability. Semi-autonomous control also substantially reduces subjective operator workload compared with full teleoperation. These findings demonstrate that effective construction-robot control should adapt human involvement to task and environmental uncertainty, using autonomy for repeatable operations and shared control for disturbance recovery.
Figures & tables
| Work | Interface | Robot | Arm(s) | Control | Skill method | Task | Subjects | Workload |
| Fu et al. [ 9 ] | DT | Excavator | N/A | Auto | Trajectory generation | Mining excavation | No | No |
| Yu et al. [ 31 ] | VR demo | KUKA KR120 | Single | LfD/Auto | HIL (GRU/VAE) | Ceiling installation | No | No |
| Ge et al. [ 10 ] | N/A | Industrial | Dual | Auto | Cooperative force control | Multi-peg assembly | No | No |
| Phung et al. [ 21 ] | VR/language/gesture | Underwater ROV | Single | SA | Supervisory control | Underwater sampling | Yes | No |
| This work | VR | FAIRINO | Dual | TeleOp/SA/Auto | Rule-based library | Drill/bracket/screw | Yes | Yes |
| Priority | Source | Behavior |
| 1 | Emergency stop / safety interlock | Immediate halt; all motion commands suppressed |
| 2 | Workspace/force limits | Autonomous contact limits override operator motion in protected states |
| 3 | Operator override (grip engaged) | Human command supersedes autonomous trajectory in shared states |
| 4 | Autonomous skill controller | Default execution when no override is active |
| Task | Subtasks Description | SAR | |
| Drill | ·Human brings the drill near the target · Robot performs precise alignment · Robot executes drilling | 1 0 0 | 0.33 |
| Bracket Handling | ·Human guides coarse motion · Robot stabilizes bracket · Robot fine-tunes bracket orientation | 1 0 0 | 0.33 |
| Screw Handling | ·Human provides coarse approach · Human triggers start/stop command · Robot aligns screw precisely and rotates | 1 1 0 | 0.66 |
| Component | Specification |
| Robot arms | 2 FAIRINO FR5 (6-DoF), controller update 125 Hz |
| End-effectors | Parallel gripper, drill head, electric screwdriver, suction gripper |
| Force/torque sensing | Six-axis sensor mounted between the end effector and task tool |
| RGB-D camera | Intel RealSense D435, 30 Hz |
| VR device | Meta Quest 3 with 6-DoF controllers |
| Workstation | Intel i9 CPU, 32 GB RAM, NVIDIA RTX 4080 |
| State | Sensors | Transition | Override | Failure |
| Target Select | RGB-D | Marker/hole found | Grip pressed | No detection |
| Coarse Move | Joint encoders | Reach pre-pose | Grip pressed | Planner fail |
| Fine Align | RGB-D, wrench | Alignment accepted | Grip pressed | Align timeout |
| Drill | Force, current, depth | Nominal depth reached or drilling terminated | E-stop | Jamming detected |
| Retract | Joint encoders | Safe retract done | Grip pressed | Collision risk |
| State | Sensors | Transition | Override | Failure |
| Pick | Gripper state | Grasp confirmed | Grip pressed | Slip |
| Flip Arm | Joint encoders | Reorient done | Grip pressed | Singularity |
| Target Select | RGB-D | Target valid | Grip pressed | No target |
| Coarse/Place | RGB-D, encoders | Placement pose | Grip pressed | Reach error |
| Fine Align | RGB-D | Alignment accepted | Grip pressed | Timeout |
| State | Sensors | Transition | Override | Failure |
| Screw Pick | Gripper | Grasp confirmed | Grip pressed | Drop |
| Flip Arm | Encoders | Reorient done | Grip pressed | Reach fail |
| Hole ID | RGB-D | Hole detected | Grip pressed | False detect |
| Coarse Approach | RGB-D | Near hole | Grip pressed | Misalignment |
| Fine Align | RGB-D, wrench | Alignment accepted | Grip pressed | Timeout |
| Fasten | F/T, depth | Nominal fastening targets reached | E-stop | Premature torque rise or jamming |
| Task | Subtasks ( ) | Human / Robot responsibility (Semi-Auto) | Mode | SAR | Outputs |
| Drilling | 5 skills | Human: select, coarse, retract; Robot: align, drill | Semi-Auto | 0.6 | Success, depth, pos., perp., time |
| Bracket | 6 skills | Human: pick, coarse, select, place; Robot: flip, align | Semi-Auto | 0.67 | Success, pos., ori., time |
| Screw | 7 skills | Human: pick, coarse, hole ID, right-arm hold; Robot: flip, align, fasten | Semi-Auto | 0.57 | Success, insertion, time |
| Mode | Success (%) | Depth (mm) | Pos. error (mm) | Perp. ( ∘ ) | Time (s) |
| TeleOp | 53 | ||||
| Semi-Auto | 82 | ||||
| Auto | 96 |
| Mode | Success (%) | Pos. Error (mm) | Ori. Error ( ∘ ) | Time (s) |
| TeleOp | 4 | |||
| Semi-Auto | 81 | |||
| Auto | 97 |
| Mode | Success (%) | Insert. Depth (mm) | Time (s) |
| TeleOp | 3 | ||
| Semi-Auto | 68 | ||
| Auto | 93 |
| Scenario | Mode | Success (%) | Depth (mm) | Pos. Error (mm) | Perpendicularity ( ∘ ) | Time (s) |
| Nominal reference | TeleOp | 53 | ||||
| Semi-Auto | 82 | |||||
| Auto | 96 | |||||
| Dim lighting | Semi-Auto | 80 | ||||
| Auto | 75 | |||||
| Dynamic colored lighting | Semi-Auto | 76 |
| Task | Mode | Mental | Physical | Temporal | Performance | Effort | Frustration | Raw TLX |
| Drilling | TeleOp | |||||||
| Semi-Auto |
| Control Mode | Binary Control Vector for Task | SAR |
| Auto | 0.0 | |
| Semi-Auto 1 | 0.2 | |
| Semi-Auto 2 | 0.4 | |
| Semi-Auto 3 | 0.6 | |
| Semi-Auto 4 | 0.8 | |
| TeleOp | 1.0 |
| Control Mode | SAR | Success (%) | Depth (mm) | Positional Error (mm) | Perpendicularity ( ∘ ) | Time (s) |
| Auto | 0.0 | 96 | 52.1 | 15.0 | 1.8 | 15.0 |
| Semi-Auto 1 | 0.2 | 92 | 51.3 | 16.8 | 2.5 | 18.3 |
| Semi-Auto 2 | 0.4 | 88 | 50.9 | 17.2 | 3.0 | 21.7 |
| Semi-Auto 3 | 0.6 | 72 | 50.6 | 20.3 | 12.2 | 32.4 |
| Semi-Auto 4 | 0.8 | 58 | 49.8 | 26.5 | 14.5 | 58.5 |
| TeleOp | 1.0 | 53 | 31.2 | 36.4 | 15.0 | 92.5 |
Human-Robot Collaboration
Human-robot collaboration studies how people and robots coordinate tasks safely, effectively, and with appropriate human control.
See all 191 papers