novtexmechМехатроника, автоматизация, управлениеMekhatronika, Avtomatizatsiya, Upravlenie1684-64272619-1253Commercial Publisher «New Technologies»10.17587/mau.25.306-314novtexmech-1569Research ArticleРОБОТЫ, МЕХАТРОНИКА И РОБОТОТЕХНИЧЕСКИЕ СИСТЕМЫROBOT, MECHATRONICS AND ROBOTIC SYSTEMSУдаленное управление промышленными роботами с использованием интерфейсов виртуальной реальностиIndustrial Robot Remote Control using Virtual Reality InterfacesЮсуповаА. Ю.IusupovaA. I.

аспирант

Санкт-Петербург

Postgraduate Student

Saint Petersburg, 194064

a.yusupova@rtc.ruТитовВ. В.TitovV. V.

науч. сотр.

Санкт-Петербург

Researcher

Saint Petersburg, 194064

СергеевА. В.SergeevA. V.

вед. инженер

Санкт-Петербург

Chief Engineer

Saint Petersburg, 194064

ЦНИИ РТКРоссияRussian State Scientific Center for Robotics and Technical CyberneticsRussian Federation202406062024256306314Copyright © Commercial Publisher «New Technologies», 20242024Commercial Publisher «New Technologies»Commercial Publisher «New Technologies»https://mech.novtex.ru/jour/about/submissions#copyrightNoticehttps://mech.novtex.ru/jour/article/view/1569

Промышленные роботы, выполняющие сложные операции, часто требуют дистанционного управления. Оператор в любое время должен иметь возможность настраивать поведение робота, задавать операции и моделировать их выполнение, синхронизировать состояние робота и его цифровой модели, менять режим управления. Интерфейсы виртуальной реальности позволяют управлять роботом в интерактивном режиме и выполнять описанные выше операции. В статье предлагается реализация системы управления на основе интерфейсов виртуальной реальности, позволяющей управлять промышленным роботом в режиме реального времени. Предлагаемое решение апробировано на двух роботах и включает в себя универсальный (итеративный) решатель обратной кинематики, планировщик траектории, планировщик задач, поддерживает работу в режимах траекторном и master-slave режимах.

Industrial robots performing complex operations often require remote control. The operator must have access to configure the robot behavior, set operations, simulate operation execution before execution, synchronize robot and its digital model state, change the control mode if necessary. Virtual reality interfaces allow to control robots interactively and perform all the operations described above. The article proposes an implementation of a control system based on virtual reality interfaces, which allows real-time control of an industrial robot. The proposed solution has been tested on two robots and includes a universal (iterative) inverse kinematics solver, a trajectory planner, a tasks scheduler, supports work in master-slave and trajectory modes.

промышленный роботвиртуальная реальностьчеловеко-машинный интерфейсуправление роботомindustrial robotvirtual realityhuman-machine interfacerobot controlИсследование выполнено при поддержке АО "Прорыв" по договору № 740/1199-Д-02 от 06.06.2023.This research is supported by the Proryv JSC under contract No. 740/1199-D-02 dated 06/06/2023.ReferencesPérez L., Diez E., Usamentiaga R., García D. F. Industrial robot control and operator training using virtual reality interfaces, Computers in Industry, 2019, no. 109, pp. 114—120.Pérez L., Diez E., Usamentiaga R., García D. F. Industrial robot control and operator training using virtual reality interfaces, Computers in Industry, 2019, no. 109, pp. 114—120.Pantelidis V. S. Reasons to use virtual reality in education and training courses and a model to determine when to use virtual reality, Themes in Science and Technology Education, 2010, no. 2(1-2), pp. 59—70.Pantelidis V. S. Reasons to use virtual reality in education and training courses and a model to determine when to use virtual reality, Themes in Science and Technology Education, 2010, no. 2(1-2), pp. 59—70.Crespo Raúl, Rene Garcia, Samuel Quiroz. Virtual Reality Application for Simulation and Off-line Programming of the Mitsubishi Movemaster RV-M1 Robot Integrated with the Oculus Rift to Improve Students Training, Procedia Computer Science, 2015, no. 75, pp. 107—112.Crespo Raúl, Rene Garcia, Samuel Quiroz. Virtual Reality Application for Simulation and Off-line Programming of the Mitsubishi Movemaster RV-M1 Robot Integrated with the Oculus Rift to Improve Students Training, Procedia Computer Science, 2015, no. 75, pp. 107—112.Moglia A., Ferrari V., Morelli L., Ferrari M., Mosca F., Cuschieri A. A Systematic review of virtual reality simulators for robot-assisted surgery, Eur. Urol., 2016, no. 69, pp. 1065—1080.Moglia A., Ferrari V., Morelli L., Ferrari M., Mosca F., Cuschieri A. A Systematic review of virtual reality simulators for robot-assisted surgery, Eur. Urol., 2016, no. 69, pp. 1065—1080.Howard M. C. A meta-analysis and systematic literature review of virtual reality rehabilitation programs, Comput. Hum. Behav., 2017, no. 70, pp. 317—327.Howard M. C. A meta-analysis and systematic literature review of virtual reality rehabilitation programs, Comput. Hum. Behav., 2017, no. 70, pp. 317—327.Lapointe J. COSMOS: a VR-based proof-of-concept interface for advanced space robot control. International Symposium on Information, Communication and Automation Technologies, 2005.Lapointe J. COSMOS: a VR-based proof-of-concept interface for advanced space robot control. International Symposium on Information, Communication and Automation Technologies, 2005.Planthaber S., Mallwitz M., Kirchner E. Immersive Robot Control in Virtual Reality to Command Robots in Space Missions, Journal of Software Engineering and Applications, 2018, no. 11, pp. 341—347, doi: 10.4236/jsea.2018.117021.Planthaber S., Mallwitz M., Kirchner E. Immersive Robot Control in Virtual Reality to Command Robots in Space Missions, Journal of Software Engineering and Applications, 2018, no. 11, pp. 341—347, doi: 10.4236/jsea.2018.117021.Chen K. et al. Fundamental development of a virtual reality simulator for four-arm disaster rescue robot OCTOPUS, 2016 IEEE International Conference on Advanced Intelligent Mechatronics (AIM), Banff, AB, Canada, 2016, pp. 721—726, doi: 10.1109/AIM.2016.7576853.Chen K. et al. Fundamental development of a virtual reality simulator for four-arm disaster rescue robot OCTOPUS, 2016 IEEE International Conference on Advanced Intelligent Mechatronics (AIM), Banff, AB, Canada, 2016, pp. 721—726, doi: 10.1109/AIM.2016.7576853.Matsas E., Batras D., Vosniakos G. Beware of the Robot: A Highly Interactive and Immersive Virtual Reality Training Application in Robotic Manufacturing Systems, Advances in Production Management Systems, 2012, vol. 397, pp. 606—613.Matsas E., Batras D., Vosniakos G. Beware of the Robot: A Highly Interactive and Immersive Virtual Reality Training Application in Robotic Manufacturing Systems, Advances in Production Management Systems, 2012, vol. 397, pp. 606—613.Wang Q., Jiao W., Yu R., Johnson M. T., Zhang Y. Modeling of Human Welders’ Operations in Virtual Reality Human—Robot Interaction, IEEE Robotics and Automation Letters, 2019, vol. 4, no. 3, pp. 2958—2964, doi: 10.1109/LRA.2019.2921928.Wang Q., Jiao W., Yu R., Johnson M. T., Zhang Y. Modeling of Human Welders’ Operations in Virtual Reality Human—Robot Interaction, IEEE Robotics and Automation Letters, 2019, vol. 4, no. 3, pp. 2958—2964, doi: 10.1109/LRA.2019.2921928.Miner N. E., Stansfield S. A. An interactive virtual reality simulation system for robot control and operator training, Proceedings of the 1994 IEEE International Conference on Robotics and Automation, 1994, vol. 2, pp. 1428—1435, doi: 10.1109/ROBOT.1994.351289.Miner N. E., Stansfield S. A. An interactive virtual reality simulation system for robot control and operator training, Proceedings of the 1994 IEEE International Conference on Robotics and Automation, 1994, vol. 2, pp. 1428—1435, doi: 10.1109/ROBOT.1994.351289.Freund E., Rossmann J. Projective virtual reality: bridging the gap between virtual reality and robotics, IEEE Transactions on Robotics and Automation, 1999, vol. 15, no. 3, pp. 411—422, doi: 10.1109/70.768175.Freund E., Rossmann J. Projective virtual reality: bridging the gap between virtual reality and robotics, IEEE Transactions on Robotics and Automation, 1999, vol. 15, no. 3, pp. 411—422, doi: 10.1109/70.768175.Freund E., Schluse M., J. Rossmann J. State oriented modeling as enabling technology for projective virtual reality, Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No.01CH37180), 2001, vol. 4, pp. 1842—1847, doi: 10.1109/IROS.2001.976341.Freund E., Schluse M., J. Rossmann J. State oriented modeling as enabling technology for projective virtual reality, Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No.01CH37180), 2001, vol. 4, pp. 1842—1847, doi: 10.1109/IROS.2001.976341.Sabater J. M., Azorin J. M., Reinoso O., Neco R. P., Garcia N. M. Dynamic virtual environment to test teleoperated systems with time delay communications. Journal of Robotic Systems, 2005, no. 22(4), pp. 167—181.Sabater J. M., Azorin J. M., Reinoso O., Neco R. P., Garcia N. M. Dynamic virtual environment to test teleoperated systems with time delay communications. Journal of Robotic Systems, 2005, no. 22(4), pp. 167—181.Tang X., Zhao D., Yamada H., Ni T. Haptic interaction in tele-operation control system of construction robot based on virtual reality, International Conference on Mechatronics and Automation, Changchun, 2009, pp. 78—83, doi: 10.1109/ICMA.2009.5246470.Tang X., Zhao D., Yamada H., Ni T. Haptic interaction in tele-operation control system of construction robot based on virtual reality, International Conference on Mechatronics and Automation, Changchun, 2009, pp. 78—83, doi: 10.1109/ICMA.2009.5246470.Tang X., Yamada H., Huang L., Ahmad A. Y. Virtual reality-based teleoperation construction robot control system with 3Dvisor device, IEEE International Conference on Mechatronics and Automation, 2010, pp. 384—388, doi: 10.1109/ICMA.2010.5588360.Tang X., Yamada H., Huang L., Ahmad A. Y. Virtual reality-based teleoperation construction robot control system with 3Dvisor device, IEEE International Conference on Mechatronics and Automation, 2010, pp. 384—388, doi: 10.1109/ICMA.2010.5588360.Matsas E., Vosniakos G. C. Design of a virtual reality training system for human—robot collaboration in manufacturing tasks. Int J Interact Des Manuf, 2017, no. 11, pp. 139—153, available at: https://doi.org/10.1007/s12008-015-0259-2.Matsas E., Vosniakos G. C. Design of a virtual reality training system for human—robot collaboration in manufacturing tasks. Int J Interact Des Manuf, 2017, no. 11, pp. 139—153, available at: https://doi.org/10.1007/s12008-015-0259-2.Malik A. A., Masood T., Bilberg A. Virtual reality in manufacturing: immersive and collaborative artificial-reality in design of humanrobot workspace, International Journal of Computer Integrated Manufacturing, 2019, vol. 33, pp. 22—37, doi: 10.1080/0951192X.2019.1690685.Malik A. A., Masood T., Bilberg A. Virtual reality in manufacturing: immersive and collaborative artificial-reality in design of humanrobot workspace, International Journal of Computer Integrated Manufacturing, 2019, vol. 33, pp. 22—37, doi: 10.1080/0951192X.2019.1690685.Wonsick M., Padir T. A Systematic Review of Virtual Reality Interfaces for Controlling and Interacting with Robots. Applied Sciences, 2020, vol. 10, no. 24, p. 9051, available at: https://doi.org/10.3390/app10249051.Wonsick M., Padir T. A Systematic Review of Virtual Reality Interfaces for Controlling and Interacting with Robots. Applied Sciences, 2020, vol. 10, no. 24, p. 9051, available at: https://doi.org/10.3390/app10249051.Robot operating system. ROS. Retrieved March 9, 2023, available at:https://www.ros.org/Robot operating system. ROS. Retrieved March 9, 2023, available at:https://www.ros.org/Unity Technologies, Unity. Retrieved March 9, 2023, available at: https://unity.com/Unity Technologies, Unity. Retrieved March 9, 2023, available at: https://unity.com/Unity Robotics Hub. Retrieved March 9, 2023, available at: https://github.com/Unity-Technologies/Unity-Robotics-HubUnity Robotics Hub. Retrieved March 9, 2023, available at: https://github.com/Unity-Technologies/Unity-Robotics-HubPérez L., Diez E., Usamentiaga R., García D. F. Industrial robot control and operator training using virtual reality interfaces, Comput. Ind., 2019, no. 109, pp. 114—120.Pérez L., Diez E., Usamentiaga R., García D. F. Industrial robot control and operator training using virtual reality interfaces, Comput. Ind., 2019, no. 109, pp. 114—120.Liu X., Nan L., Lin Y., Han J., Liu J., Ku T. Data and model hybrid-driven virtual reality robot operating system, Frontiers in Energy Research, 2022, vol. 10, doi: 10.3389/fenrg.2022.1101887.Liu X., Nan L., Lin Y., Han J., Liu J., Ku T. Data and model hybrid-driven virtual reality robot operating system, Frontiers in Energy Research, 2022, vol. 10, doi: 10.3389/fenrg.2022.1101887.Garg G., Kuts V., Anbarjafari G. Digital Twin for FANUC Robots: Industrial Robot Programming and Simulation Using Virtual Reality. Sustainability, 2021,vol. 13, no.18, p. 10336, available at: https://doi.org/10.3390/su131810336.Garg G., Kuts V., Anbarjafari G. Digital Twin for FANUC Robots: Industrial Robot Programming and Simulation Using Virtual Reality. Sustainability, 2021,vol. 13, no.18, p. 10336, available at: https://doi.org/10.3390/su131810336.Burghardt A., Szybicki D., Gierlak P., Kurc K., Pietruś P., Cygan R. Programming of Industrial Robots Using Virtual Reality and Digital Twins, Appl. Sci., 2020, no. 10, p. 486, available at: https://doi.org/10.3390/app10020486.Burghardt A., Szybicki D., Gierlak P., Kurc K., Pietruś P., Cygan R. Programming of Industrial Robots Using Virtual Reality and Digital Twins, Appl. Sci., 2020, no. 10, p. 486, available at: https://doi.org/10.3390/app10020486.Gallala A., Kumar A. A., Hichri B., Plapper P. Digital Twin for Human—Robot Interactions by Means of Industry 4.0 Enabling Technologies, Sensors, 2022, vol. 22, p. 4950, available at: https://doi.org/10.3390/s22134950.Gallala A., Kumar A. A., Hichri B., Plapper P. Digital Twin for Human—Robot Interactions by Means of Industry 4.0 Enabling Technologies, Sensors, 2022, vol. 22, p. 4950, available at: https://doi.org/10.3390/s22134950.Togias T., Gkournelos C., Angelakis P., Michalos G., Makris S. Virtual reality environment for industrial robot control and path design, Procedia CIRP, 2021, no.100, pp. 133—138.Togias T., Gkournelos C., Angelakis P., Michalos G., Makris S. Virtual reality environment for industrial robot control and path design, Procedia CIRP, 2021, no.100, pp. 133—138.Chitta S., Marder-Eppstein E., Meeussen et. al. Ros_control: A generic and simple control framework for ROS, J. Open Source Softw., 2017, no. 2, pp. 456.Chitta S., Marder-Eppstein E., Meeussen et. al. Ros_control: A generic and simple control framework for ROS, J. Open Source Softw., 2017, no. 2, pp. 456.Russian Federation Patent No. RU2785920C1, 2022.Russian Federation Patent No. RU2785920C1, 2022.Sergeev A., Sergeev S. Complexity reduction of interfaces of robotic and ergatic systems, Robotics and Technical Cybernetics, 2019, no. 7(2), pp. 109—118, available at: https://doi.org/10.31776/RTCJ.7204.Sergeev A., Sergeev S. Complexity reduction of interfaces of robotic and ergatic systems, Robotics and Technical Cybernetics, 2019, no. 7(2), pp. 109—118, available at: https://doi.org/10.31776/RTCJ.7204.Sergeev A., Titov V. Shardyko I. Induced virtual environment for control of a manipulator designed for working with radioactive materials, Robotics and Technical Cybernetics, 2021, no. 9(1), pp. 32—41, available at: https://doi.org/10.31776/RTCJ.9104.Sergeev A., Titov V. Shardyko I. Induced virtual environment for control of a manipulator designed for working with radioactive materials, Robotics and Technical Cybernetics, 2021, no. 9(1), pp. 32—41, available at: https://doi.org/10.31776/RTCJ.9104.

The authors declare that there are no conflicts of interest present.