novtexmechМехатроника, автоматизация, управлениеMekhatronika, Avtomatizatsiya, Upravlenie1684-64272619-1253Commercial Publisher «New Technologies»10.17587/mau.24.533-541novtexmech-1441Research ArticleРОБОТЫ, МЕХАТРОНИКА И РОБОТОТЕХНИЧЕСКИЕ СИСТЕМЫROBOT, MECHATRONICS AND ROBOTIC SYSTEMSЗахват неизвестных объектов с помощью автономного манипулятора: современное состояние, проблемы и перспективыGrasping of Unknown Objects with an Autonomous Manipulator: State of the Art, Problems and ProspectsВоронковА. Д.VoronkovA. D.

аспирант

г. Москва

Postgraduate Student

Moscow, 119454

a.voronkov.rtu@yandex.ruДианеС. А. К.DianeS. A. K.

г. Москва

Moscow, 119454

МИРЭА — Российский технологический университетРоссияMIREA — Russian Technological UniversityRussian Federation2023091020232410533541Copyright © Commercial Publisher «New Technologies», 20232023Commercial Publisher «New Technologies»Commercial Publisher «New Technologies»https://mech.novtex.ru/jour/about/submissions#copyrightNoticehttps://mech.novtex.ru/jour/article/view/1441

Для выполнения практических задач, стоящих перед современной робототехникой, требуется разработка подходов к захвату незнакомых объектов, поскольку в условиях реального мира робот сталкивается с большим их разнообразием. Подходы, подразумевающие наличие полной информации об объектах рабочей области (3D-модель, массогабаритные характеристики), не практичны и могут быть использованы только в контролируемых условиях, таких как работа на конвейере c типовыми деталями. Поэтому научное сообщество и ряд отраслей промышленности заинтересованы в исследовании методов, повышающих способность робота адаптироваться к новым, незнакомым условиям.В данной статье приводится подборка основных направлений в задачах визуального анализа сцены и захвата неизвестных объектов манипуляционным роботом. Рассмотрены отличия существующих подходов по различным критериям, преимущества и недостатки имеющихся решений. Статья может быть полезна для ознакомления с предметной областью.

To fulfill the practical needs of modern robotics, it is necessary to develop approaches for grasping unknown objects, since in the real world the robot faces a large variety of them. Approaches that imply the availability of complete information about the objects of the working area (3D model, weight and size characteristics) are not practical and can only be used in controlled conditions, such as working on a conveyor with standard details. Therefore, the scientific community and a number of industries are interested in research methods that increase the robot’s ability to adapt to new, unfamiliar conditions. This article presents main problems and research directions in the field of visual scene perception and grasping unknown objects by a manipulative robot. We discuss the differences in existing approaches according to various criteria, as well as advantages and disadvantages of existing solutions. The article may be useful to get acquainted with the subject area.

захват незнакомых объектовизбегание столкновенийманипуляционный роботмашинное об- учениезахват объектов статической сценывосприятие незнакомых объектовunknown objects graspingcollision avoidancemanipulative robotmachine learninggrasping objects of static sceneunknown objects perceptionReferencesLiu J., Balatti P., Ellis K., Hadjivelichkov D., Stoyanov D., Ajoudani A., Kanoulas D. Garbage collection and sorting with a mobile manipulator using deep learning and whole-body control, 2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), Munich, Germany, 2021, pp. 408—414.Liu J., Balatti P., Ellis K., Hadjivelichkov D., Stoyanov D., Ajoudani A., Kanoulas D. Garbage collection and sorting with a mobile manipulator using deep learning and whole-body control, 2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), Munich, Germany, 2021, pp. 408—414.Zhang H., Liang H., Ni T., Huang L. Research on multiobject sorting system based on deep learning, Sensors, 2021, vol. 21, no. 18, 21 p.Zhang H., Liang H., Ni T., Huang L. Research on multiobject sorting system based on deep learning, Sensors, 2021, vol. 21, no. 18, 21 p.Guerin J., Stephane T., Nyiri E., Gibaru O. Unsupervised robotic sorting: towards autonomous decision making robots, International Journal of Artificial Intelligence & Applications, 2018, vol. 9, pp. 81—98.Guerin J., Stephane T., Nyiri E., Gibaru O. Unsupervised robotic sorting: towards autonomous decision making robots, International Journal of Artificial Intelligence & Applications, 2018, vol. 9, pp. 81—98.Bormann R., Brito B., Lindermayr J., Omainska M., Patel M. Towards automated order picking robots for warehouses and retail, Computer Vision Systems. ICVS 2019. Lecture Notes in Computer Science, 2019, vol. 11754, Springer, Cham.Bormann R., Brito B., Lindermayr J., Omainska M., Patel M. Towards automated order picking robots for warehouses and retail, Computer Vision Systems. ICVS 2019. Lecture Notes in Computer Science, 2019, vol. 11754, Springer, Cham.Habibian S., Dadvar M., Peykari B. Design and implementation of a maxi-sized mobile robot (Karo) for rescue missions, Robomech, 2021, vol. 8, no. 1, 33 p.Habibian S., Dadvar M., Peykari B. Design and implementation of a maxi-sized mobile robot (Karo) for rescue missions, Robomech, 2021, vol. 8, no. 1, 33 p.Murali A., Mousavian A., Eppner C., Paxton C., Fox D. 6-DOF grasping for target-driven object manipulation in clutter, ICRA, 2020 (virtual), 2020, 7 p.Murali A., Mousavian A., Eppner C., Paxton C., Fox D. 6-DOF grasping for target-driven object manipulation in clutter, ICRA, 2020 (virtual), 2020, 7 p.Mousavian A., Eppner C., Fox D. 6-DOF GraspNet: variational grasp generation for object manipulation, 2019 IEEE/ CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 2901—2910.Mousavian A., Eppner C., Fox D. 6-DOF GraspNet: variational grasp generation for object manipulation, 2019 IEEE/ CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 2901—2910.Sauvet B., Levesque F., Park S., Cardou P. Model-based grasping of unknown objects from a random pile, Robotics, 2019, vol. 8, no. 3, 18 p.Sauvet B., Levesque F., Park S., Cardou P. Model-based grasping of unknown objects from a random pile, Robotics, 2019, vol. 8, no. 3, 18 p.Sundermeyer M., Mousavian A., Triebel R., Fox D. Contact-GraspNet: efficient 6-DoF grasp generation in cluttered scenes, 2021 IEEE International Conference on Robotics and Automation (ICRA), Xian, China, 2021, pp. 13438—13444.Sundermeyer M., Mousavian A., Triebel R., Fox D. Contact-GraspNet: efficient 6-DoF grasp generation in cluttered scenes, 2021 IEEE International Conference on Robotics and Automation (ICRA), Xian, China, 2021, pp. 13438—13444.Huang B., Han S., Boularias A., Yu J. DIPN: Deep interaction prediction network with application to clutter removal, Conference: 2021 IEEE International Conference on Robotics and Automation (ICRA), Xian, China — 2021, pp. 4694—4701.Huang B., Han S., Boularias A., Yu J. DIPN: Deep interaction prediction network with application to clutter removal, Conference: 2021 IEEE International Conference on Robotics and Automation (ICRA), Xian, China — 2021, pp. 4694—4701.Zhang H., Lan X., Bai S., Zhou X., Tian Z., Zheng N. ROI-based robotic grasp detection for object overlapping scenes, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China — 2019, pp. 4768—4775.Zhang H., Lan X., Bai S., Zhou X., Tian Z., Zheng N. ROI-based robotic grasp detection for object overlapping scenes, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China — 2019, pp. 4768—4775.Qin Y., Chen R., Zhu H., Song M., Xu J., Su H. S4G: amodal single-view single-shot SE(3) grasp detection in cluttered scenes, 3rd Conference on Robot Learning (CoRL 2019), Osaka, Japan, 2019, pp. 53—65.Qin Y., Chen R., Zhu H., Song M., Xu J., Su H. S4G: amodal single-view single-shot SE(3) grasp detection in cluttered scenes, 3rd Conference on Robot Learning (CoRL 2019), Osaka, Japan, 2019, pp. 53—65.Breyer M., Chung J., Ott L., Siegwart R., Nieto J. Volumetric grasping network: real-time 6 DOF grasp detection in clutter, 4th Conference on Robot Learning (CoRL 2020), 2020, pp. 1602—1611.Breyer M., Chung J., Ott L., Siegwart R., Nieto J. Volumetric grasping network: real-time 6 DOF grasp detection in clutter, 4th Conference on Robot Learning (CoRL 2020), 2020, pp. 1602—1611.Pas A., Gualtieri M., Saenko K., Platt R. Grasp pose detection in point clouds, The International Journal of Robotics Research, 2017, vol. 36, pp. 1455—1473.Pas A., Gualtieri M., Saenko K., Platt R. Grasp pose detection in point clouds, The International Journal of Robotics Research, 2017, vol. 36, pp. 1455—1473.Lei Q., Meijer J., Wisse M. A survey of unknown object grasping and our fast grasping algorithm-C shape grasping, 2017 3rd International Conference on Control, Automation and Robotics (ICCAR), Nagoya, Japan, 2017, pp. 150—157.Lei Q., Meijer J., Wisse M. A survey of unknown object grasping and our fast grasping algorithm-C shape grasping, 2017 3rd International Conference on Control, Automation and Robotics (ICCAR), Nagoya, Japan, 2017, pp. 150—157.Abhijit M., Federico T., Perez-Gracia A. Grasping unknown objects in clutter by superquadric representation, 2018 Second IEEE International Conference on Robotic Computing (IRC), California, USA, 2018, pp. 292—299.Abhijit M., Federico T., Perez-Gracia A. Grasping unknown objects in clutter by superquadric representation, 2018 Second IEEE International Conference on Robotic Computing (IRC), California, USA, 2018, pp. 292—299.Herzog A., Pastor P., Kalakrishnan M., Righetti L., Bohg J., Asfour T., Schaal S. Learning of grasp selection based on shape-templates, Autonomous Robots, 2014, vol. 36, pp. 51—65.Herzog A., Pastor P., Kalakrishnan M., Righetti L., Bohg J., Asfour T., Schaal S. Learning of grasp selection based on shape-templates, Autonomous Robots, 2014, vol. 36, pp. 51—65.Kappler D., Bohg J., Schaal S. Leveraging big data for grasp planning, IEEE International Conference on Robotics and Automation (ICRA), 2015, pp. 4304—4311.Kappler D., Bohg J., Schaal S. Leveraging big data for grasp planning, IEEE International Conference on Robotics and Automation (ICRA), 2015, pp. 4304—4311.Savvin S. V., Sirota A. A. Superpixel segmentation methods and their application for image analysis with heterogeneous texture, Vestnik VGU, seriya: sistemnyj analiz i informacionnye tekhnologii, 2016, no. 4, pp. 165—173 (in Russian).Savvin S. V., Sirota A. A. Superpixel segmentation methods and their application for image analysis with heterogeneous texture, Vestnik VGU, seriya: sistemnyj analiz i informacionnye tekhnologii, 2016, no. 4, pp. 165—173 (in Russian).Danielczuk M., Matl M., Gupta S., Li A., Lee A., Mahler J., Goldberg K. Segmenting unknown 3D objects from real depth images using Mask R-CNN trained on synthetic data, 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 7283—7290.Danielczuk M., Matl M., Gupta S., Li A., Lee A., Mahler J., Goldberg K. Segmenting unknown 3D objects from real depth images using Mask R-CNN trained on synthetic data, 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 7283—7290.Xiang Y., Xie C., Mousavian A., Fox D. Learning RGB-D feature embeddings for unseen object instance segmentation, 4th Conference on Robot Learning (CoRL 2020), 2020, pp. 461—470.Xiang Y., Xie C., Mousavian A., Fox D. Learning RGB-D feature embeddings for unseen object instance segmentation, 4th Conference on Robot Learning (CoRL 2020), 2020, pp. 461—470.Xie C., Xiang Y., Mousavian A., Fox D. Unseen Object Instance Segmentation for Robotic Environments, IEEE Transactions on Robotics, vol. 37, no. 5, pp. 1343—1359.Xie C., Xiang Y., Mousavian A., Fox D. Unseen Object Instance Segmentation for Robotic Environments, IEEE Transactions on Robotics, vol. 37, no. 5, pp. 1343—1359.Back S., Lee J., Kim T., Noh S., Kang R., Bak S., Lee K. Unseen object amodal instance segmentation via hierarchical occlusion modeling, 2022 International Conference on Robotics and Automation (ICRA), Philadelphia, PA, US, 2022, pp. 5085—5092.Back S., Lee J., Kim T., Noh S., Kang R., Bak S., Lee K. Unseen object amodal instance segmentation via hierarchical occlusion modeling, 2022 International Conference on Robotics and Automation (ICRA), Philadelphia, PA, US, 2022, pp. 5085—5092.Charles R., Su H., Kaichun M.,Guibas L. PointNet: Deep learning on point sets for 3D classification and segmentation, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 77—85.Charles R., Su H., Kaichun M.,Guibas L. PointNet: Deep learning on point sets for 3D classification and segmentation, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 77—85.Charles R., Yi L., Su H., Leonidas J. PointNet++: deep hierarchical feature learning on point sets in a metric space, Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17), Curran Associates Inc., Red Hook, NY, USA, 2017, pp. 5105—5114.Charles R., Yi L., Su H., Leonidas J. PointNet++: deep hierarchical feature learning on point sets in a metric space, Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17), Curran Associates Inc., Red Hook, NY, USA, 2017, pp. 5105—5114.Leskov A. G., Seliverstova E. V. Algorithm for planning and selecting a method for grasping a deformable object with a multi-fingered gripper of a manipulative robot, Mekhatronika, Avtomatizaciya, Upravlenie, 2017, vol. 18, no. 11, pp. 739—744 (in Russian).Leskov A. G., Seliverstova E. V. Algorithm for planning and selecting a method for grasping a deformable object with a multi-fingered gripper of a manipulative robot, Mekhatronika, Avtomatizaciya, Upravlenie, 2017, vol. 18, no. 11, pp. 739—744 (in Russian).Mais Zh. Application of convolution neural networks to train a manipulative robot to capture objects, Politekhnicheskij molodezhnyj zhurnal, 2020, no. 1, 5 p. (in Russian).Mais Zh. Application of convolution neural networks to train a manipulative robot to capture objects, Politekhnicheskij molodezhnyj zhurnal, 2020, no. 1, 5 p. (in Russian).Lenz I., Lee H., Saxena A. Deep learning for detecting robotic grasps, The International Journal of Robotics Research, 2013, no. 34, pp. 705—724.Lenz I., Lee H., Saxena A. Deep learning for detecting robotic grasps, The International Journal of Robotics Research, 2013, no. 34, pp. 705—724.Redmon J., Angelova A. Real-time grasp detection using convolutional neural networks, Proceedings — IEEE International Conference on Robotics and Automation, 2015, pp. 1316—1322.Redmon J., Angelova A. Real-time grasp detection using convolutional neural networks, Proceedings — IEEE International Conference on Robotics and Automation, 2015, pp. 1316—1322.Li T., Wang F., Ru C., Jiang Y., Li J. Keypoint-based robotic grasp detection scheme in multi-object scenes, Sensors, 2021, vol. 21, no. 6, 15 p.Li T., Wang F., Ru C., Jiang Y., Li J. Keypoint-based robotic grasp detection scheme in multi-object scenes, Sensors, 2021, vol. 21, no. 6, 15 p.Abdullov T. S., Muhametov A. N. Robotic grasping of objects of known shape using a monocular camera, Fizikomatematicheskie, estestvenno-nauchnye i social’nye aspekty sovremennogo razvitiya nauki, tekhniki i obshchestva. Materialy I gorodskoj molodyozhnoj nauchnoj konferencii Fiziko-matematicheskie, estestvenno-nauchnye i social’nye aspekty sovremennogo razvitiya nauki, tekhniki i obshchestva. Materialy I gorodskoj molodyozhnoj nauchnoj konferencii, Kazan — 2021, pp. 3—6 (in Russian).Abdullov T. S., Muhametov A. N. Robotic grasping of objects of known shape using a monocular camera, Fizikomatematicheskie, estestvenno-nauchnye i social’nye aspekty sovremennogo razvitiya nauki, tekhniki i obshchestva. Materialy I gorodskoj molodyozhnoj nauchnoj konferencii Fiziko-matematicheskie, estestvenno-nauchnye i social’nye aspekty sovremennogo razvitiya nauki, tekhniki i obshchestva. Materialy I gorodskoj molodyozhnoj nauchnoj konferencii, Kazan — 2021, pp. 3—6 (in Russian).Agboh W., Dogar M. Real-time online re-planning for grasping under clutter and uncertainty, 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), 2018, 8 p.Agboh W., Dogar M. Real-time online re-planning for grasping under clutter and uncertainty, 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), 2018, 8 p.Firman M., Aodha O., Julier S., Gabriel J. Structured prediction of unobserved voxels from a single depth image, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 5431—5440.Firman M., Aodha O., Julier S., Gabriel J. Structured prediction of unobserved voxels from a single depth image, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 5431—5440.Rawan A., Al Akkad M. A. Machine learning in robotic grasping tasks: a survey, Priborostroenie v XXI veke — 2020. Integraciya nauki, obrazovaniya i proizvodstva. Cbornik materialov XVI Vserossijskoj nauchno-tekhnicheskoj konferencii, Izhevsk, 2021, pp. 164—170.Rawan A., Al Akkad M. A. Machine learning in robotic grasping tasks: a survey, Priborostroenie v XXI veke — 2020. Integraciya nauki, obrazovaniya i proizvodstva. Cbornik materialov XVI Vserossijskoj nauchno-tekhnicheskoj konferencii, Izhevsk, 2021, pp. 164—170.Boularias A., Bagnell J. A. Stentz A. Learning to manipulate unknown objects in clutter by reinforcement, AAAI’15: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015, pp. 1336—1342.Boularias A., Bagnell J. A. Stentz A. Learning to manipulate unknown objects in clutter by reinforcement, AAAI’15: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015, pp. 1336—1342.Zeng A., Song S., Welker S., Lee J., Rodriguez A., Funkhouser T. Learning synergies between pushing and grasping with selfsupervised deep reinforcement learning, IEEE International Conference on Intelligent Robots and Systems (IROS), 2018, pp. 4238—4245.Zeng A., Song S., Welker S., Lee J., Rodriguez A., Funkhouser T. Learning synergies between pushing and grasping with selfsupervised deep reinforcement learning, IEEE International Conference on Intelligent Robots and Systems (IROS), 2018, pp. 4238—4245.Zhang Z., Gu J. Grasp planning of 3D objects using genetic algorithm, 2012 IEEE International Conference on Automation and Logistics, Zhengzhou, China, 2012, pp. 646—651.Zhang Z., Gu J. Grasp planning of 3D objects using genetic algorithm, 2012 IEEE International Conference on Automation and Logistics, Zhengzhou, China, 2012, pp. 646—651.Gabellieri C., Angelini F., Arapi V., Palleschi A., Catalano M., Grioli G., Pallottino L., Bicchi A., Bianchi M., Garabini M. Grasp it like a pro: grasp of unknown objects with robotic hands based on skilled human expertise, IEEE Robotics and Automation Letters, 2020, vol. 5, no. 2, pp. 2808—2815.Gabellieri C., Angelini F., Arapi V., Palleschi A., Catalano M., Grioli G., Pallottino L., Bicchi A., Bianchi M., Garabini M. Grasp it like a pro: grasp of unknown objects with robotic hands based on skilled human expertise, IEEE Robotics and Automation Letters, 2020, vol. 5, no. 2, pp. 2808—2815.Palmieri G., Scoccia C. Motion Planning and Control of Redundant Manipulators for Dynamical Obstacle Avoidance, Machines, 2021, vol. 9, no. 6—121.Palmieri G., Scoccia C. Motion Planning and Control of Redundant Manipulators for Dynamical Obstacle Avoidance, Machines, 2021, vol. 9, no. 6—121.Wei K., Ren B. A Method on Dynamic Path Planning for Robotic Manipulator Autonomous Obstacle Avoidance Based on an Improved RRT Algorithm, Sensors, 2018, vol. 18, no. 2—571.Wei K., Ren B. A Method on Dynamic Path Planning for Robotic Manipulator Autonomous Obstacle Avoidance Based on an Improved RRT Algorithm, Sensors, 2018, vol. 18, no. 2—571.Ibraheem B., Ibrahim A., Talib A. Motion Planning for a Robot Arm by Using Genetic Algorithm, Jordan Journal of Mechanical and Industrial Engineering, 2008, vol. 2, no. 3, pp. 131—136;Ibraheem B., Ibrahim A., Talib A. Motion Planning for a Robot Arm by Using Genetic Algorithm, Jordan Journal of Mechanical and Industrial Engineering, 2008, vol. 2, no. 3, pp. 131—136;Leskov A. G., Bazhinova K. V., Seliverstova E. V. Methods for determining the quality of object grasping, Vestnik MGTU im. N. E. Baumana. Ser. Mashinostroenie, 2017, no. 3 (114), pp. 122—139 (in Russian).Leskov A. G., Bazhinova K. V., Seliverstova E. V. Methods for determining the quality of object grasping, Vestnik MGTU im. N. E. Baumana. Ser. Mashinostroenie, 2017, no. 3 (114), pp. 122—139 (in Russian).Bicchi A. On the closure properties of robotic grasping, The International Journal of Robotics Research, 1995, vol. 27, no. 14, pp. 319—334.Bicchi A. On the closure properties of robotic grasping, The International Journal of Robotics Research, 1995, vol. 27, no. 14, pp. 319—334.Gualtieri M., Pas A., Saenko K., Platt R. High precision grasp pose detection in dense clutter, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea (South), 2016, pp. 598—605.Gualtieri M., Pas A., Saenko K., Platt R. High precision grasp pose detection in dense clutter, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea (South), 2016, pp. 598—605.

The authors declare that there are no conflicts of interest present.