Abstract
Transparent objects are widely used in our daily lives, e.g., glass cups, plastic bottles and glass pan lids in the kitchen. They are also common in research laboratories, e.g., vials, glass flasks and Petri dishes. Many of these objects are fragile and easy to break, therefore need to be handled with extra attention. To have robots work in such environments, robots need to have safe interaction with transparent objects. Without this capability, transparent objects may be broken or have their contents spilled. Broken glass or spilled liquid will pose hazards to the robot and the people that share its space.However, the transparent materials of these objects violate the Lambertian assumption that optical 3D sensors (e.g., LiDAR and RGB-D cameras) are based on. The Lambertian assumption assumes that objects reflect light evenly in all directions, resulting in a uniform surface brightness from all viewing angles. However, the surfaces of transparent objects both reflect and refract light, which breaks the Lambertian assumption. This property makes obtaining accurate depth data from depth sensors challenging, and the data is either invalid or contains unpredictable noise, further complicating the perception of transparent objects.
Humans can use multiple sensory information to mitigate the uncertainty during manipulation of challenging objects. For instance, humans utilise the sense of touch to confirm and refine the information initially gathered by vision. This tactile feedback allows for adjustments of the hand’s pose, ensuring more accurate and secure manipulation. The goal of this thesis is to address the inherent challenges associated with robotic perception and manipulation of transparent objects by using multiple modalities, i.e., vision and tactile sensing.
This thesis first investigates how to achieve transparent object perception and manipulation using only RGB-D vision. A method named A4T, i.e., Affordance for Transparent object depth reconstruction and manipulation, is proposed. It couples depth reconstruction and manipulation planning for the manipulation of transparent objects via a visual affordance map that represents the object functionality of each pixel. The affordance map can improve depth reconstruction via a multi-step reconstruction approach and also output the functionalities of object regions for manipulation.
Secondly, it studies how to integrate both vision and tactile sensing so that robots can better localise the positions of transparent objects and enhance the grasping success rate. Vision is first used to predict the horizontal upper regions named poking regions, where the robot can poke the object to obtain a good tactile reading while leading to minimal disturbance to the object’s status. Then, given the local profiles improved with tactile sensing, a heuristic grasp is planned for grasping the transparent object. By leveraging tactile sensing, robots could reduce the reliance on accurate depth information for grasping transparent objects.
Finally, it studies the grasping of thin and flexible transparent objects, i.e., plastic papers. Inspired by the rotation mechanism in paper feeder machines and money counters, a novel rotatable tactile sensor named RoTip is first developed. This sensor is capable o sensing the all-around fingertip area and actively rotating its body. Then, a novel visual-tactile framework for handling thin, flexible transparent objects is designed. Specifically, RGB-D vision is used to locate the grasping position, and tactile sensing is used to adjust the end-effector pose to ensure sufficient contact during handling.
In conclusion, the thesis has contributed to advances in perceiving and manipulating different transparent objects, such as plastic cups, glass mugs, and plastic papers. This is the first study that shows the reliability of combining global information from vision and accurate local information from tactile sensing for transparent object perception and manipulation. This breakthrough has the potential to benefit the robotics community, enabling more nuanced and effective interactions with a broad range of materials and environments.
Date of Award | 1 Sept 2024 |
---|---|
Original language | English |
Awarding Institution |
|
Supervisor | Toktam Mahmoodi (Supervisor) & Thanh-Toan Do (Supervisor) |