cs.ROMar 20, 2026

An Open Source Computer Vision and Machine Learning Framework for Affordable Life Science Robotic Automation

Authors: Zachary Logan, Andrew Dudash, Daniel Negrón

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present an open-source robotic framework that integrates computer vision and machine learning based inverse kinematics to enable low-cost laboratory automation tasks such as colony picking and liquid handling. The system uses a custom trained U-net model for semantic segmentation of microbial cultures, combined with Mixture Density Network for predicating joint angles of a simple 5-DOF robot arm. We evaluated the framework using a modified robot arm, upgraded with a custom liquid handling end-effector. Experimental results demonstrate the framework's feasibility for precise, repeatable operations, with mean positional error below 1 mm and joint angle prediction errors below 4 degrees and colony detection capabilities with IoU score of 0.537 and Dice coefficient of 0.596.

Figures & tables

Fig. 1: To automate the colony detection and sampling process, we developed a low-cost and open source based robotic platform integrating a modified 5-DOF robot based on the design in [ 45 ] chosen for its ease of assembly, readily available parts and affordability. The arm was fitted with upgraded Hitec D645MW and Towerpro MG92B servos for improved precision. At the end-effector, we attached an open-source digital pipette adapted from Yoshikawa et al. [ 47 ] with a 10 mL syringe, enabling the precise aspiration and dispenstion of liquids and a high resolution SVPRO 8 MP USB camera. To control the servo motors we used an Arduino Uno microcontroller [ 48 ] because of its popularity in robots, expansive documentation, open design, and low cost.
Fig. 2: The Flow chart illustrates the control architecture for the automated colony picking system. The robot first hovers over the Petri dish and captures an image using the camera. The image is passed through the trained U-net segmentation model, if any colonies are detected, the center points of the contours bounding the colonies are extracted and converted from pixel coordinates to world coordinates. These coordinates are passed through the MDN-based IK model to compute the required joint angles. Lastly the robot arm moves to the picking point to acquire the sample. If no colonies are detected, the system outputs a message stating that nothing was found and all samples have been collected.
Fig. 3: Representative samples from the agar plate image dataset used for training and testing the U-net segmentation model. a) A homemade agar plate using beef bouillon, sugar and gelatin, which produced slightly clear areas along the outer edge due to evaporation of some of the water used in production. b) A standard laboratory prepared agar plate using conventional techniques and materials, resulting in much cleaner and richer bacterial colony growth.
Fig. 4: Example outputs from the semantic segmentation model on two agar plate images. (Left): Original Input image of an agar plate; (Right) Corresponding predicted segmentation mask. The green circles indicate the center point of the detected colonies and indicate the input used for the downstream IK calculations. a). Homemade agar plate prepared using beef bouillon, sugar and gelatin. b). Standard laboratory prepared agar plate.

Explore similar work

CardsList

  1. Scalable Low-Cost Laboratory Automation: A Digital Twin-Integrated Robotic Platform for Autonomous Liquid Handling (RAINBOT)

    Jul 22, 2026Mohamed Rami Ayeche, Souhil Sid, Ahyen Mostofa +3Laboratory AutomationRobotic Manipulation

    Acceleration Consortium, University of Toronto, Toronto, Canada. · AISCIA Informatics, Qatar Science and Technology Park, Doha, Qatar · Hamad Bin Khalifa University (HBKU), Education City, Doha, Qatar.

  2. Learning From a Steady Hand: A Weakly Supervised Agent for Robot Assistance under Microscopy

    Jan 28, 2026Huanyu Tian, Martin Huber, Lingyun Zeng +7Robotic ManipulationAssistive Robotics

    H. Tian, M. Huber, Z. Han, L. Zeng, T. Vercauteren, and C. Bergeles are with the School of Biomedical Engineering & Imaging Sciences, King’s College London, UK. · W. Bennett, G. Silvestri, G. Mendizabal-Ruiz, C. Bergeles, and A. Chavez-Badiola are with Conceivable Life Sciences, New York City, US and London, UK. · A. Chavez-Badiola is with Hope IVF Mexico, Mexico.

  3. Utilizing Inpainting for Keypoint Detection for Vision-Based Control of Robotic Manipulators

    Apr 14, 2026Sreejani Chatterjee, Venkatesh Mullur, Abhinav Gandhi +1Vision-Based Robot ControlKeypoint Detection

    Robotics Engineering Program, Worcester Polytechnic Institute, MA, USA