novtexmechМехатроника, автоматизация, управлениеMekhatronika, Avtomatizatsiya, Upravlenie1684-64272619-1253Commercial Publisher «New Technologies»10.17587/mau.22.634-643novtexmech-1097Research ArticleРОБОТЫ, МЕХАТРОНИКА И РОБОТОТЕХНИЧЕСКИЕ СИСТЕМЫROBOT, MECHATRONICS AND ROBOTIC SYSTEMSРаспределенная система локализации объектов в рабочей зоне модульного реконфигурируемого мобильного роботаDistributed System for Objects Localization in the Working Area of a Modular Reconfigurable Mobile RobotВолковаМ. А.VolkovaM. A.

Старший преподаватель

Volkova Maria A. Senior Lecturer

volkova_m@mirea.ruРомановА. М.RomanovA. M.

Кандидат технических наук, доцент

Associate Professor

romanov@mirea.ruРомановМ. П.RomanovM. P.

Доктор технических наук, профессор

Director of the Institute of Cybernetics

m_romanov@mirea.ruМИРЭА — Российский технологический университетРоссияMIREA — Russian technological universityRussian Federation2021011220212212634643Copyright © Commercial Publisher «New Technologies», 20212021Commercial Publisher «New Technologies»Commercial Publisher «New Technologies»https://mech.novtex.ru/jour/about/submissions#copyrightNoticehttps://mech.novtex.ru/jour/article/view/1097

Предлагается новый подход к локализации объектов в рабочей зоне модульного реконфигурируемого робота (МРР), предполагающий установку роботом стационарных опорных измерительных пунктов (ОИП), состоящих из отделяемых от него модулей. В основе данного подхода лежит ранее предложенная авторами архитектура системы управления МРР и новый метод сопоставления информации о скорости и положении объектов, получаемой с различных сенсоров. Суть подхода состоит в следующем: прибыв в зону выполнения задачи, МРР выделяет из своего состава ОИП, содержащий источник питания, вычислительное устройство, беспроводной приемопередатчик и сенсор. ОИП осуществляют слежение за рабочей зоной с использованием разнородных датчиков (камер, дальномеров и т. д.), сегментацию объектов по получаемым измерениям и передачу этой информации на борт робота. Далее с использованием нового метода сопоставления информации о скорости и положении объектов происходит сопоставление измерений, полученных с различных сенсоров, что позволяет локализовать объекты даже в тех случаях, когда они не видны для части ОИП. Одним из ключевых преимуществ нового подхода является возможность его реализации в распределенной архитектуре МРР. Проведенные модельные экспериментальные исследования показали, что по критерию качества отслеживания Multiple Object Tracking Accuracy (MOTA) метод имеет оценку 86 %, что превосходит большинство известных аналогов, а динамическая ошибка локализации объектов в рабочей зоне 8Ѕ7 м с использованием двух камер и одного дальномера не превышает 10 см.

The paper proposes a novel approach to the objects localization in the working area of a modular reconfigurable robot (MRR), which implies the installation of stationary monitoring points (SMP), consisting of detachable robot’s modules and in- stalled by robot itself. This approach is based on the architecture of the MRR control system previously proposed by the authors and a new method for comparing information about the speed and position obtained from various sensors. The key steps of the approach are following. Upon arriving in the target area, the MRR places SMPs, which consist of a power source, a computing device, a wireless transceiver and a sensor, detached from the robot. Then SMPs monitor the working area using different types of sensors (cameras, rangefinders, etc.), perform segmentation of the measured data and transfer this information to the robot. Further a sensor fusion is performed using a novel object tracking method, which makes it possible to localize target objects even in those cases when they are not visible by some of the SMPs. One of the key advantages of the new approach is a possibility of implementation in the distributed architecture of a MRR. The simulation results show that proposed method has Multiple Object Tracking Accuracy (MOTA) metric of 86 %, which is higher than the most of its analogues, while the estimated dynamic object localization error in a 8x7 m working area using 2 cameras and 1 rangefinder does not exceed 10 cm.

реконфигурируемые роботылокализация объектовслежение за объектамикомплексирование сенсорной информацииreconfigurable robotsobject localizationobject trackingsensor fusionИсследование выполнено при финансовой поддержке РФФИ в рамках научного проекта № 19-38-90301.The reported study was funded by RFBR, project number 19-38-90301.ReferencesBillard A., Kragic D. Trends and challenges in robot manipulation // Science. 2019. Vol. 364, N. 6446. P. eaat8414.Billard A., Kragic D. Trends and challenges in robot manipulation, Science, 2019, vol. 364, no. 6446, pp. eaat8414.Yasuda Y. D. V., Martins L. E. G., Cappabianco F. A. M. Autonomous visual navigation for mobile robots: A systematic literature review // ACM Computing Surveys (CSUR). 2020. Vol. 53, N. 1. P. 1—34.Yasuda Y. D. V., Martins L. E. G., Cappabianco F. A. M. Autonomous visual navigation for mobile robots: A systematic literature review, ACM Computing Surveys (CSUR), 2020, vol. 53, no. 1, pp. 1—34.Tolani V. et al. Visual navigation among humans with optimal control as a supervisor // IEEE Robotics and Automation Letters. 2021. Vol. 6, N. 2. P. 2288—2295.Tolani V. et al. Visual navigation among humans with optimal control as a supervisor, IEEE Robotics and Automation Letters, 2021, vol. 6, no. 2, pp. 2288—2295.Payá L., Gil A., Reinoso O. A state-of-the-art review on mapping and localization of mobile robots using omnidirectional vision sensors // Journal of Sensors. 2017. Vol. 2017. P. 1—20.Payá L., Gil A., Reinoso O. A state-of-the-art review on mapping and localization of mobile robots using omnidirectional vision sensors, Journal of Sensors, 2017, vol. 2017, pp. 1—20.Chen H. et al. Vision and laser fused SLAM in indoor environments with multi-robot system // Assembly Automation. 2019. Vol. 39, N. 2. P. 297—307.Chen H. et al. Vision and laser fused SLAM in indoor environments with multi-robot system, Assembly Automation, 2019, vol. 39, no. 2, pp. 297—307.Tang B., Jiang L. Binocular stereovision omnidirectional motion handling robot // International Journal of Advanced Robotic Systems. 2020. Vol. 17, N. 3. P. 1729881420926852.Tang B., Jiang L. Binocular stereovision omnidirectional motion handling robot, International Journal of Advanced Robotic Systems, 2020, vol. 17, no. 3, pp. 1729881420926852.Kuo C. Y. et al. Development of an immersive SLAMbased VR system for teleoperation of a mobile manipulator in an unknown environment // Computers in Industry. 2021. Vol. 132. P. 103502.Kuo C. Y. et al. Development of an immersive SLAM-based VR system for teleoperation of a mobile manipulator in an unknown environment, Computers in Industry, 2021, vol. 132, pp. 103502.Romanov A. M. et al. A Navigation System for Intelligent Mobile Robots // 2019 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus). IEEE, 2019. P. 652—656.Romanov A. M. et al. A Navigation System for Intelligent Mobile Robots, 2019 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), IEEE, 2019, pp. 652—656.Yang Y. et al. 3D multiview basketball players detection and localization based on probabilistic occupancy // 2018 Digital Image Computing: Techniques and Applications (DICTA). IEEE, 2018. P. 1—8.Yang Y. et al. 3D multiview basketball players detection and localization based on probabilistic occupancy, 2018 Digital Image Computing: Techniques and Applications (DICTA), IEEE, 2018, pp. 1—8.Егорцев М. В., Диане С. А. К., Кац Н. Д. Алгоритмическое обеспечение системы внешнего наблюдения и маршрутизации автономных мобильных роботов // Российский технологический журнал. 2021. Т. 9, № 3. С. 15—23.Egortsev M. V., Diane S. K., Kaz N. D. Algorithmic support of the system of external observation and routing of autonomous mobile robots, Russian Technological Journal, 2021, vol. 9, no. 3, pp. 15—23 (in Russian).Ben Y., Cengiz K. Research on Visual Orientation Guidance of Industrial Robot Based on CAD Model under Binocular Vision // Computer-Aided Design and Applications. 2022. Vol. 19. N. S2. P. 52—63.Ben Y., Cengiz K. Research on Visual Orientation Guidance of Industrial Robot Based on CAD Model under Binocular Vision, Computer-Aided Design and Applications, 2022, vol. 19, no. S2, pp. 52—63.Srigrarom S. et al. Multi-camera Multi-drone Detection, Tracking and Localization with Trajectory-based Re-identification // 2021 Second International Symposium on Instrumentation, Control, Artificial Intelligence, and Robotics (ICA-SYMP). IEEE, 2021. P. 1—6.Srigrarom S. et al. Multi-camera Multi-drone Detection, Tracking and Localization with Trajectory-based Re-identification, 2021 Second International Symposium on Instrumentation, Control, Artificial Intelligence, and Robotics (ICA-SYMP), IEEE, 2021, pp. 1—6.Park S. et al. Survey on Anti-Drone Systems: Components, Designs, and Challenges // IEEE Access. 2021. Vol. 9. P. 42635—42659.Park S. et al. Survey on Anti-Drone Systems: Components, Designs, and Challenges, IEEE Access, 2021, vol. 9, pp. 42635—42659.Khalyasmaa A. I. et. al. Diagnostic system for OHL state assessment // 2015 International Siberian Conference on Control and Communications (SIBCON). IEEE, 2015. P. 1—5.Khalyasmaa A. I. et. al. Diagnostic system for OHL state assessment, 2015 International Siberian Conference on Control and Communications (SIBCON), IEEE, 2015, pp. 1—5.Khalyasmaa A. I. et. al. Robotic intelligence laboratory for overhead transmission lines assessment // 2016 57th International Scientific Conference on Power and Electrical Engineering of Riga Technical University (RTUCON). IEEE, 2016. P. 1—6.Khalyasmaa A. I. et. al. Robotic intelligence laboratory for overhead transmission lines assessment, 2016 57th International Scientific Conference on Power and Electrical Engineering of Riga Technical University (RTUCON), IEEE, 2016, pp. 1—6.Schuster M. J. et al. Distributed stereo vision-based 6D localization and mapping for multi-robot teams // Journal of Field Robotics. 2019. Vol. 36, N. 2. P. 305—332.Schuster M. J. et al. Distributed stereo vision-based 6D localization and mapping for multi-robot teams, Journal of Field Robotics, 2019, vol. 36, no. 2, pp. 305—332.Андреев В. П., Подураев Ю. В. Функционально-модульный принцип построения гетерогенных мобильных роботов // Экстремальная робототехника. 2016. Т. 1, № 1. С. 39—49.Andreev V. P., Poduraev Yu. V. Functional-modular design of heterogeneous mobile robotic systems, Extreme Robotics, 2016, vol. 1, no. 1, pp. 39—49 (in Russian).Romanov A. M., Romanov M. P., Shestakov E. I. A novel architecture for control systems of modular reconfigurable robots // 2017 IEEE II International Conference on Control in Technical Systems (CTS). IEEE, 2017. P. 131—134.Romanov A. M., Romanov M. P., Shestakov E. I. A novel architecture for control systems of modular reconfigurable robots, 2017 IEEE II International Conference on Control in Technical Systems (CTS), IEEE, 2017, pp. 131—134.Yao R. et al. Video object segmentation and tracking: A survey // ACM Transactions on Intelligent Systems and Technology (TIST). 2020. Vol. 11, N. 4. P. 1—47.Yao R. et al. Video object segmentation and tracking: A survey, ACM Transactions on Intelligent Systems and Technology (TIST), 2020, vol. 11, no. 4, pp. 1—47.Li Y., Ibanez-Guzman J. Lidar for autonomous driving: The principles, challenges, and trends for automotive lidar and perception systems // IEEE Signal Processing Magazine. 2020. Vol. 37, N. 4. P. 50—61.Li Y., Ibanez-Guzman J. Lidar for autonomous driving: The principles, challenges, and trends for automotive lidar and perception systems, IEEE Signal Processing Magazine, 2020, vol. 37, no. 4, pp. 50—61.Minaee S. et al. Image segmentation using deep learning: A survey //IEEE Transactions on Pattern Analysis and Machine Intelligence. 2021.Minaee S. et al. Image segmentation using deep learning: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.Ariff S. A. M. et al. Exploratory Study of 3d Point Cloud Triangulation for Smart City Modelling and Visualization // The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences. 2020. Vol. 44. P. 71—79.Ariff S. A. M. et al. Exploratory Study of 3d Point Cloud Triangulation for Smart City Modelling and Visualization, The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2020, vol. 44, pp. 71—79.Álvarez H. et al. A Multi Camera and Multi Laser Calibration Method for 3D Reconstruction of Revolution Parts // Sensors. 2021. Vol. 21, N. 3. P. 765.Álvarez H. et al. A Multi Camera and Multi Laser Calibration Method for 3D Reconstruction of Revolution Parts, Sensors, 2021, vol. 21, no. 3, pp. 765.Llamazares Á., Molinos E. J., Oca a M. Detection and tracking of moving obstacles (DATMO): a review // Robotica. 2020. Vol. 38, N. 5. P. 761—774.Llamazares Á., Molinos E. J., Oca@ña M. Detection and tracking of moving obstacles (DATMO): a review, Robotica, 2020, vol. 38, no. 5, pp. 761—774.Zhang Z. et al. Multiple target tracking based on multiple hypotheses tracking and modified ensemble Kalman filter in multi-sensor fusion // Sensors. 2019. Vol. 19, N. 14. P. 3118.Zhang Z. et al. Multiple target tracking based on multiple hypotheses tracking and modified ensemble Kalman filter in multi-sensor fusion, Sensors, 2019, vol. 19, no. 14, pp. 3118.Sekii T. Robust, real-time 3d tracking of multiple objects with similar appearances // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016. P. 4275—4283.Sekii T. Robust, real-time 3d tracking of multiple objects with similar appearances, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4275—4283.Sun Q. et al. Multiple object tracking for yellow feather broilers based on foreground detection and deep learning // INMATEH-Agricultural Engineering. 2019. Vol. 58, N. 2.Sun Q. et al. Multiple object tracking for yellow feather broilers based on foreground detection and deep learning, INMATEHAgricultural Engineering, 2019, vol. 58, no. 2.Мокшин В. В., Кирпичников А. П., Шарнин Л. М. Отслеживание объектов в видеопотоке по значимым признакам на основе фильтрации частиц // Вестник Казанского технологического университета. 2013. Vol. 16, N. 18. P. 306—310.Mokshin V. V., Kirpichnikov A. P., Sharnin L. M. Particle filtering-based objects tracking in a video stream using significant features, Vestnik Kazanskogo tehnologicheskogo universiteta, 2013, vol. 16, no. 18, pp. 297—303 (in Russian).Zhou J., Kwan C. Tracking of multiple pixel targets using multiple cameras // International Symposium on Neural Networks. Springer, Cham, 2018. P. 484—493.Zhou J., Kwan C. Tracking of multiple pixel targets using multiple cameras, International Symposium on Neural Networks, Springer, Cham, 2018, pp. 484—493.Eshel R., Moses Y. Homography based multiple camera detection and tracking of people in a dense crowd // 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2008. P. 1—8.Eshel R., Moses Y. Homography based multiple camera detection and tracking of people in a dense crowd, 2008 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2008, pp. 1—8.Teizer J., Caldas C. H., Haas C. T. Real-time threedimensional occupancy grid modeling for the detection and tracking of construction resources // Journal of Construction Engineering and Management. 2007. Vol. 133, N. 11. P. 880—888.Teizer J., Caldas C. H., Haas C. T. Real-time three-dimensional occupancy grid modeling for the detection and tracking of construction resources, Journal of Construction Engineering and Management, 2007, vol. 133, no. 11, pp. 880—888.Coué C. et al. Bayesian occupancy filtering for multitarget tracking: an automotive application // The International Journal of Robotics Research. 2006. Vol. 25, N. 1. P. 19—30.Coué C. et al. Bayesian occupancy filtering for multitarget tracking: an automotive application, The International Journal of Robotics Research, 2006, vol. 25, no. 1, pp. 19—30.Gindele T. et al. Bayesian occupancy grid filter for dynamic environments using prior map knowledge // 2009 IEEE Intelligent Vehicles Symposium. IEEE, 2009. P. 669—676.Gindele T. et al. Bayesian occupancy grid filter for dynamic environments using prior map knowledge, 2009 IEEE Intelligent Vehicles Symposium, IEEE, 2009, pp. 669—676.Nuss D., Yuan T., Krehl G., Stuebler M., Reuter S., Dietmayer K. Fusion of laser and radar sensor datawith a sequential Monte Carlo Bayesian occupancy filter // Proceedings of the 2015 IEEE Intelligent VehiclesSymposium (IV), Seoul, Korea, 28 June—1 July 2015. P. 1074—1081.Nuss D.,Yuan T.,Krehl G., Stuebler M., Reuter S., Dietmayer K. Fusion of laser and radar sensor datawith a sequential Monte Carlo Bayesian occupancy filter, Proceedings of the 2015 IEEE Intelligent VehiclesSymposium (IV), Seoul, Korea, 28 June—1 July 2015, pp. 1074—1081.Taj M., Cavallaro A. Distributed and decentralized multicamera tracking // IEEE Signal Processing Magazine. 2011. Vol. 28, N. 3. P. 46—58.Taj M., Cavallaro A. Distributed and decentralized multicamera tracking, IEEE Signal Processing Magazine, 2011, vol. 28, no. 3, pp. 46—58.Taj M., Cavallaro A. Simultaneous Detection and Tracking with Multiple Cameras // Machine Learning for Computer Vision. Springer, Berlin, Heidelberg, 2013. С. 197—214.Taj M., Cavallaro A. Simultaneous Detection and Tracking with Multiple Cameras, Machine Learning for Computer Vision, Springer, Berlin, Heidelberg, 2013, pp. 197—214.Liang Q. et al. Multi-Player Tracking for Multi-View Sports Videos with Improved K-Shortest Path Algorithm // Applied Sciences. 2020. Vol. 10, N. 3. P. 864.Liang Q. et al. Multi-Player Tracking for Multi-View Sports Videos with Improved K-Shortest Path Algorithm, Applied Sciences, 2020, vol. 10, no. 3, pp. 864.Romanov A., Yashunskiy V., Chiu W.-Y. SABER: Modular Reconfigurable Robot for Industrial Applications // IEEE 17th International Conference on Automation Science and Engineering (CASE) 2021. 2021 (в печати).Romanov A., Yashunskiy V., Chiu W.-Y. SABER: Modular Reconfigurable Robot for Industrial Applications, IEEE 17th International Conference on Automation Science and Engineering (CASE). 2021, 2021 (in press).Romanov A. M., Mikheenko I. S. A novel approach for creating modular reconfigurable robots with distributed power system // 2018 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus). IEEE, 2018. P. 974—978.Romanov A. M., Mikheenko I. S. A novel approach for creating modular reconfigurable robots with distributed power system, 2018 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), IEEE, 2018, pp. 974—978.Romanov A., Slepynina E. Real-time Ethernet POWERLINK Communication for ROS. Part I. General Concept // 2020 Ural Smart Energy Conference (USEC). IEEE, 2020. P. 159—162.Romanov A., Slepynina E. Real-time Ethernet POWERLINK Communication for ROS. Part I. General Concept, 2020 Ural Smart Energy Conference (USEC). IEEE, 2020, pp. 159—162.Romanov A., Slepynina E. Real-time Ethernet POWERLINK Communication for ROS. Part II. Hardware and Software // 2020 Ural Smart Energy Conference (USEC). IEEE, 2020. P. 163—166.Romanov A., Slepynina E. Real-time Ethernet POWERLINK Communication for ROS. Part II. Hardware and Software, 2020 Ural Smart Energy Conference (USEC), IEEE, 2020, pp. 163—166.Romanov A. M., Gringoli F., Sikora A. A precise synchronization method for future wireless TSN networks // IEEE Transactions on Industrial Informatics. 2021. Vol. 17, N. 5. P. 3682—3692.Romanov A. M., Gringoli F., Sikora A. A precise synchronization method for future wireless TSN networks, IEEE Transactions on Industrial Informatics, 2021, vol. 17, no. 5, pp. 3682—3692.Romanov A. M. et al. Modular Reconfigurable Robot Distributed Computing System for Tracking Multiple Objects // IEEE Systems Journal. 2021. Vol. 15, N. 1. P. 802—813.Romanov A. M. et al. Modular Reconfigurable Robot Distributed Computing System for Tracking Multiple Objects, IEEE Systems Journal, 2021, vol. 15, no. 1, pp. 802—813.Rossi R. J. Mathematical statistics: an introduction to likelihood based inference. John Wiley & Sons, 2018.Rossi R. J. Mathematical statistics: an introduction to likelihood based inference, John Wiley & Sons, 2018.Hu Z. et al. Extrinsic calibration of 2-D laser rangefinder and camera from single shot based on minimal solution // IEEE Transactions on Instrumentation and Measurement. 2016. Vol. 65, N. 4. P. 915—929.Hu Z. et al. Extrinsic calibration of 2-D laser rangefinder and camera from single shot based on minimal solution, IEEE Transactions on Instrumentation and Measurement, 2016, vol. 65, no. 4, pp. 915—929.Yang C., Duraiswami R., Davis L. Fast multiple object tracking via a hierarchical particle filter // Tenth IEEE International Conference on Computer Vision (ICCV’05). IEEE, 2005. Vol. 1. P. 212—219.Yang C., Duraiswami R., Davis L. Fast multiple object tracking via a hierarchical particle filter, Tenth IEEE International Conference on Computer Vision (ICCV’05), IEEE, 2005, vol. 1, pp. 212—219.Kong L. et al. Online Multiple Athlete Tracking with Pose-Based Long-Term Temporal Dependencies // Sensors. 2021. Vol. 21, N. 1. P. 197.Kong L. et al. Online Multiple Athlete Tracking with Pose-Based Long-Term Temporal Dependencies, Sensors, 2021, vol. 21, no. 1, pp. 197.

The authors declare that there are no conflicts of interest present.