A Passive AI System for Verifying Physical State on Automated Liquid Handlers
Authors: Junqiong Joanne Qiu, Zeckria Kamrany, Ananya Anand, Emma Vidal, Yunjia Johanna Bai, Nathan S Chen, Sean Son, Varada Abhyankar, Taryn Jakub, Eleazar Eskin
Organizations: Department of Computational Medicine, David Geffen School of Medicine, UCLA, Los Angeles, CA, USA · Department of Computer Science, UCLA, Los Angeles, CA, USA · Department of Mathematics, UCLA, Los Angeles, CA, USA
Automated liquid handlers execute digital protocols, but operators must assemble the deck and confirm that the physical setup matches the intended experiment. We present the Labware Setup Checker, a passive vision system that verifies deck preparation on an Opentrons Flex without modifying its hardware or firmware. A consumer webcam mounted outside the robot supplies images to a browser application. The checker segments the deck into slots, corrects slot assignments using the known deck geometry, classifies the labware, and compares the observed setup with a reference protocol library to report missing, misplaced, or unexpected labware. It operates independently of the robot controller and does not interrupt or gate a run. During SwabSeq Respiratory Viral Panel (RVP) pre-PCR deck preparation, the checker tracked a slot from empty to a base and then to the completed plate-and-base assembly, identifying the completed assembly in all 29 frames before robot motion. Across these frames, 95.1% of the 348 labware classifications for slots requiring exact matches agreed with the protocol reference. In 480 frames with partially occluded reservoirs, all 6,720 slot assignments were correct. Across three camera positions and two lighting conditions, slot assignment was 100% correct, and 97.73% of labware classifications matched the protocol reference. Median verification time was 2.10 s on a laptop using only its CPU and 4.24 s from frame capture to receipt of results during AWS deployment in the laboratory. These results show that an external camera and independent software can provide verification of deck setup against protocol requirements within seconds, allowing operators to review the physical setup before starting a run.
Figures & tables
Figure 1: Web interface and protocol-aware deck verification. A. The checker displays the live camera feed with predicted slot and labware labels, status updates, detection logs, and the predicted protocol. B. An incorrect pre-PCR deck setup in which B3 is empty and the expected 50 µL tip rack is placed at C3. C. The returned verification result. Green boxes indicate that the observed labware matches the protocol or that an expected-empty slot is empty. The red box at B3 indicates that the expected tip rack is missing, and the orange box at C3 indicates that the tip rack was detected in the wrong slot.
Figure 2: Recognition of an empty deck and tracking of C2 during SwabSeq Respiratory Viral Panel pre-PCR preparation. A. Representative frame from the empty-deck session, in which only the permanent C1 temperature module and D1 aluminum block were present. B. C2 was empty when recording began. The red outline indicates that the expected assembly was not yet present. C. The completed MicroAmp plate-and-base assembly at C2. The green outline indicates that the observed assembly matched the protocol. D. C2 predictions across all 37 frames collected before robot motion. Each marker represents one analyzed frame. The checker classified C2 as empty for seven frames, as the MicroAmp base in the next frame, and as the complete MicroAmp plate-and-base assembly in all subsequent 29 frames.
Figure 3: The checker under alternate Flex camera views. A–B. Two alternate camera viewpoints during live use of the checker.
Figure 4: Training dynamics of the perception models. A. Segmentation validation performance measured by mask mAP@50 and mAP@50–95 across 50 epochs. B. Segmentation training and validation loss, calculated as the sum of bounding-box, segmentation, classification, and distribution focal losses. C. Classification validation performance measured by top-1 and top-5 accuracy across 19 recorded epochs. D. Classification training and validation loss. Both models converged within the logged training runs, reaching high validation performance while validation loss remained low.
Figure 5: Laboratory deployment of the checker. A consumer webcam (yellow box) mounted beside the Opentrons Flex observes the deck, while the browser interface runs on an adjacent computer. The camera and computer operate outside the robot controller and require no modification to the Flex hardware or firmware.
Figure 6: Dataset composition for model development. Panel A shows the 85/10/5 train/validation/test split and the threefold augmentation applied to the training split only; teal columns are training data and gold/orange columns are held-out validation and test data. Counts are full-deck images for segmentation datasets and cropped labware images for classification datasets. Panel B shows original, pre-augmentation Flex classifier cropped training images for labware classes used by the prePCR and postPCR protocols. Empty slots and rare non-protocol classes are not included in Panel B.
Figure 7: Computer vision and protocol-matching pipeline. A camera frame is normalized, segmented into deck slots, geometrically corrected to the canonical Flex deck, classified with masked and cropped labware images, compared against the reference protocol, and returned to the browser as an annotated interface.
Figure 8: Checker communication flow. The browser frontend manages the user session, camera stream, periodic frame analysis, protocol setup requests, log polling, keep-alive messages, and session release. The FastAPI backend creates the session lease, runs the vision and protocol-matching pipeline, builds logs, renews the lease, and frees backend state when the user closes the checker.
a Data Science and Learning Division, Argonne National Laboratory, Lemont, IL 60439, USA. · b Department of Computer Science, University of Chicago, Chicago, IL 60637, USA.
Key Laboratory of Smart Manufacturing in Energy Chemical Process Ministry of Education, East China University of Science and Technology, Shanghai, CN. · Department of Computer Science and Engineering, East China University of Science and Technology, Shanghai, CN. · AI for Science Center, Shanghai AI Laboratory, Shanghai, CN. +2
Acceleration Consortium, University of Toronto, Toronto, Canada. · AISCIA Informatics, Qatar Science and Technology Park, Doha, Qatar · Hamad Bin Khalifa University (HBKU), Education City, Doha, Qatar.