novtexmechМехатроника, автоматизация, управлениеMekhatronika, Avtomatizatsiya, Upravlenie1684-64272619-1253Commercial Publisher «New Technologies»10.17587/mau.23.507-514novtexmech-1253Research ArticleСИСТЕМНЫЙ АНАЛИЗ, УПРАВЛЕНИЕ И ОБРАБОТКА ИНФОРМАЦИИSYSTEM ANALYSIS, CONTROL AND INFORMATION PROCESSINGМетод мультиагентного обучения с подкреплением в системах с переменным числом агентовMethod of Multi-Agent Reinforcement Learning in Systems with a Variable Number of AgentsПетренкоВ. И.PetrenkoV. I.

 канд. техн. наук, зав. кафедрой организации и технологии защиты информации 

Petrenko Vyacheslav I., Cand. of Tech. Sc., Head of the department of organization and technology of information security

Stavropol, 355017

vipetrenko@ncfu.ruТебуеваФ. Б.TebuevaF. B.

 д-р физ.-мат. наук, зав. кафедрой компьютерной безопасности 

ftebueva@ncfu.ruГурчинскийМ. М.GurchinskyM. M.

 аспирант, программист лаборатории робототехнических систем 

gurcmikhail@yandex.ruПавловА. С.PavlovA. S.

 инженер-лаборант кафедры компьютерной безопасности 

anspavlov@ncfu.ruСеверо-Кавказский федеральный университетРоссияNorth-Caucasus Federal UniversityRussian Federation2022091020222310507514Copyright © Commercial Publisher «New Technologies», 20222022Commercial Publisher «New Technologies»Commercial Publisher «New Technologies»https://mech.novtex.ru/jour/about/submissions#copyrightNoticehttps://mech.novtex.ru/jour/article/view/1253

Методы мультиагентного обучения с подкреплением являются одним из новейших и активно развивающихся направлений машинного обучения. Среди методов мультиагентного обучения с подкреплением одним из наиболее перспективных является метод MADDPG, достоинством которого является высокая сходимость процесса обучения. Недостатком метода MADDPG является необходимость обеспечения равенства числа агентов N на стадии обучения и числа агентов K на стадии функционирования. В то же время целевые мультиагентные системы (МАС), такие как группы БПЛА или мобильных наземных роботов, являются системами с переменным числом агентов, что не позволяет применять в них метод MADDPG. Для решения данной проблемы в статье предложен усовершенствованный метод MADDPG для мультиагентного обучения с подкреплением в системах с переменным числом агентов. Усовершенствованный метод MADDPG базируется на гипотезе о том, что для выполнения своих функций агенту нужна информация о состоянии не всех прочих агентов МАС, а только нескольких ближайших соседей. На основе данной гипотезы предложен метод гибридного совместного/независимого обучения МАС с переменным числом агентов, который предполагает обучение некоторого небольшого числа агентов N для обеспечения функционирования произвольного числа агентов K, K > N. Проведенные эксперименты показали, что усовершенствованный метод MADDPG обеспечивает сопоставимую с оригинальным методом эффективность функционирования МАС при варьировании числа K агентов на стадии функционирования в широких пределах.

Multi-agent reinforcement learning methods are one of the newest and actively developing areas of machine learning. Among the methods of multi-agent reinforcement learning, one of the most promising is the MADDPG method, the advantage of which is the high convergence of the learning process. The disadvantage of the MADDPG method is the need to ensure the equality of the number of agents N at the training stage and the number of agents K at the functioning stage. At the same time, target multi-agent systems (MAS), such as groups of UAVs or mobile ground robots, are systems with a variable number of agents, which does not allow the use of the MADDPG method in them. To solve this problem, the article proposes an improved MADDPG method for multi-agent reinforcement learning in systems with a variable number of agents. The improved MADDPG method is based on the hypothesis that to perform its functions, an agent needs information about the state of not all other MAS agents, but only a few nearest neighbors. Based on this hypothesis, a method of hybrid joint / independent learning of MAS with a variable number of agents is proposed, which involves training a small number of agents N to ensure the functioning of an arbitrary number of agents K, K> N. The experiments have shown that the improved MADDPG method provides an efficiency of MAS functioning com-parable to the original method with varying the number of K agents at the stage of functioning within wide limits.

мультиагентные системымашинное обучениемультиагентное обучение с подкреплениемсовместное обучениенезависимое обучениепеременное число агентовmulti-agent systemsmachine learningmulti-agent reinforcement learningcollaborative learningindependent learningvariable number of agentsReferencesKovács G., Yussupova N., Rizvanov D. Resource management simulation using multi-agent approach and semantic constraints, Pollack Period, 2017, vol. 12, no. 1, pp. 45—58.Kovács G., Yussupova N., Rizvanov D. Resource management simulation using multi-agent approach and semantic constraints, Pollack Period, 2017, vol. 12, no. 1, pp. 45—58.Darintsev O., Migranov A. Task Distribution Module for a Team of Robots Based on Genetic Algorithms: Synthesis Methodology and Testing, Proceedings of the 21st International Conference; Complex Systems: Control and Modeling Problems, CSCMP 2019, 2019, pp. 296—300.Darintsev O., Migranov A. Task Distribution Module for a Team of Robots Based on Genetic Algorithms: Synthesis Methodology and Testing, Proceedings of the 21st International Conference; Complex Systems: Control and Modeling Problems, CSCMP 2019, 2019, pp. 296—300.Darintsev O. V. et al. Methods of a heterogeneous multiagent robotic system group control, Procedia Computer Science, 2019, vol. 150, pp. 687—694.Darintsev O. V. et al. Methods of a heterogeneous multiagent robotic system group control, Procedia Computer Science, 2019, vol. 150, pp. 687—694.Wang L., Törngren M., Onori M. Current status and advancement of cyber-physical systems in manufacturing, Journal of Manufacturing Systems, 2015, vol. 37, pp. 517—527.Wang L., Törngren M., Onori M. Current status and advancement of cyber-physical systems in manufacturing, Journal of Manufacturing Systems, 2015, vol. 37, pp. 517—527.Munasypov R. A., Masalimov K. A. Neural network models for diagnostics of complex technical objects state by example of electrochemical treatment process, Proceedings of the 2nd International Ural Conference on Measurements, UralCon 2017, 2017, pp. 156—160.Munasypov R. A., Masalimov K. A. Neural network models for diagnostics of complex technical objects state by example of electrochemical treatment process, Proceedings of the 2nd International Ural Conference on Measurements, UralCon 2017, 2017, pp. 156—160.Bonilla Venegas F. V., Moya M. J., Litvin A., Lukyanov E., Marín Pillajo L. E. Modeling and Simulation of the Robot Mitsubishi RV-2JA controlled by electromyographic signals, Enfoque UTE, vol. 9 (2), pp. 208—222.Bonilla Venegas F. V., Moya M. J., Litvin A., Lukyanov E., Marín Pillajo L. E. Modeling and Simulation of the Robot Mitsubishi RV-2JA controlled by electromyographic signals, Enfoque UTE, vol. 9 (2), pp. 208—222.Vokhmintcev A. V., Melnikov A. V., Mironov K. V., Burlutsky V. V. Reconstruction of Three-Dimensional Maps Based on Closed-Form Solutions of the Variational Problem of Multisensor Data Registration, Reports of the Academy of Sciences, 2019, vol. 484, no. 6, pp. 672—677.Vokhmintcev A. V., Melnikov A. V., Mironov K. V., Burlutsky V. V. Reconstruction of Three-Dimensional Maps Based on Closed-Form Solutions of the Variational Problem of Multisensor Data Registration, Reports of the Academy of Sciences, 2019, vol. 484, no. 6, pp. 672—677.Bogdanov A., Dudorov E., Permyakov A., Pronin A., Kutlubaev I. Control system of a manipulator of the anthropomorphic robot FEDOR, Proceedings of the International Conference on Developments in eSystems Engineering, DeSE, 2019, pp. 449—454.Bogdanov A., Dudorov E., Permyakov A., Pronin A., Kutlubaev I. Control system of a manipulator of the anthropomorphic robot FEDOR, Proceedings of the International Conference on Developments in eSystems Engineering, DeSE, 2019, pp. 449—454.Petrenko V., Tebueva F., Antonov V., Untewsky N., Gurchinsky M. Energy-Efficient Path Planning: Designed Software Implementation, Proceedings of the 21st International Workshop on Computer Science and Information Technologies (CSIT 2019), 2019, vol. 3, pp. 112—118.Petrenko V., Tebueva F., Antonov V., Untewsky N., Gurchinsky M. Energy-Efficient Path Planning: Designed Software Implementation, Proceedings of the 21st International Workshop on Computer Science and Information Technologies (CSIT 2019), 2019, vol. 3, pp. 112—118.Bogdanov M., Nasyrov D., Dumchikova I., Samigullin A. Processing of Biomedical Data with Machine Learning, Proceedings of the 21st International Workshop on Computer Science and Information Technologies (CSIT 2019), 2019, vol. 3, pp. 6—16.Bogdanov M., Nasyrov D., Dumchikova I., Samigullin A. Processing of Biomedical Data with Machine Learning, Proceedings of the 21st International Workshop on Computer Science and Information Technologies (CSIT 2019), 2019, vol. 3, pp. 6—16.Petrenko V., Tebueva F., Gurchinsky M., Antonov V. A Robotic Complex Control Method Based on Deep Reinforcement Learning of Recurrent Neural Networks for Automatic Harvesting of Greenhouse Crops, Proceedings of the 8th scientific Conference on Information Technologies For Intelligent Decision Making Support (ITIDS 2020), 2020, vol. 174, pp. 340—346.Petrenko V., Tebueva F., Gurchinsky M., Antonov V. A Robotic Complex Control Method Based on Deep Reinforcement Learning of Recurrent Neural Networks for Automatic Harvesting of Greenhouse Crops, Proceedings of the 8th scientific Conference on Information Technologies For Intelligent Decision Making Support (ITIDS 2020), 2020, vol. 174, pp. 340—346.Petrenko V., Tebueva F., Antonov V., Gurchinsky M., Ryabtsev S., Burianov A. Cooperative Motion Planning Method for Two Anthropomorphic Manipulators, Proceedings of the 7th Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS 2019), 2019, vol. 166, pp. 146—151.Petrenko V., Tebueva F., Antonov V., Gurchinsky M., Ryabtsev S., Burianov A. Cooperative Motion Planning Method for Two Anthropomorphic Manipulators, Proceedings of the 7th Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS 2019), 2019, vol. 166, pp. 146—151.Petrenko V., Tebueva F., Pavlov A., Antonov V., Kochanov M. Path Planning Method in the Formation of the Configuration of a Multifunctional Modular Robot Using a Swarm Control Strategy, Proceedings of the 7th Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS 2019), 2019, vol. 166, pp. 165—170.Petrenko V., Tebueva F., Pavlov A., Antonov V., Kochanov M. Path Planning Method in the Formation of the Configuration of a Multifunctional Modular Robot Using a Swarm Control Strategy, Proceedings of the 7th Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS 2019), 2019, vol. 166, pp. 165—170.Mnih V. et al. Human-level control through deep reinforcement learning, Nature, 2015, vol. 518, pp. 529—533.Mnih V. et al. Human-level control through deep reinforcement learning, Nature, 2015, vol. 518, pp. 529—533.Petrenko V., Tebueva F., Pavlov A., Svistunov N. Machine Learning Algorithm for Anthropomorphic Manipulator Control System, Proceedings of the 8th Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS 2020), 2020, vol. 174, pp. 353—358.Petrenko V., Tebueva F., Pavlov A., Svistunov N. Machine Learning Algorithm for Anthropomorphic Manipulator Control System, Proceedings of the 8th Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS 2020), 2020, vol. 174, pp. 353—358.Hernandez-Leal P., Kartal B., Taylor M. E. A survey and critique of multiagent deep reinforcement learning, Autonomous Agents and Multi-Agent Systems, 2019, vol. 33, pp. 750—797.Hernandez-Leal P., Kartal B., Taylor M. E. A survey and critique of multiagent deep reinforcement learning, Autonomous Agents and Multi-Agent Systems, 2019, vol. 33, pp. 750—797.Pshikhopov V., Medvedev M., Medvedeva T. Terminal Motion Control of Multicopter Group, Proceedings of the 4th International Conference on Control and Robotics Engineering, ICCRE 2019, 2019, pp. 1—6.Pshikhopov V., Medvedev M., Medvedeva T. Terminal Motion Control of Multicopter Group, Proceedings of the 4th International Conference on Control and Robotics Engineering, ICCRE 2019, 2019, pp. 1—6.Wang H., Zhao H., Ma D., Wei J. Cyber Physical System Framework for UAV Communications, Electrical Engineering and Systems Science, 2020, pp. 1—41.Wang H., Zhao H., Ma D., Wei J. Cyber Physical System Framework for UAV Communications, Electrical Engineering and Systems Science, 2020, pp. 1—41.Yusupova N., Rizvanov D., Andrushko D. Cyber-Physical Systems and Reliability Issues, Proceedings of the 8th Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS 2020), 2020, vol. 174, pp. 133—137.Yusupova N., Rizvanov D., Andrushko D. Cyber-Physical Systems and Reliability Issues, Proceedings of the 8th Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS 2020), 2020, vol. 174, pp. 133—137.Fabarisov T.,Yusupova N., Ding K., Morozov A., Janschek K. Model-based stochastic error propagation analysis for cyberphysical systems, Acta Polytechnica Hungarica, 2020, vol. 17, no. 8, pp. 15—28.Fabarisov T.,Yusupova N., Ding K., Morozov A., Janschek K. Model-based stochastic error propagation analysis for cyberphysical systems, Acta Polytechnica Hungarica, 2020, vol. 17, no. 8, pp. 15—28.Valiev E., Yusupova N., Morozov A., Janschek K., Beyer M. Evaluation of the Impact of Random Computing Hardware Faults on the Performance of Convolutional Neural Networks, Proceedings of the 8th Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS 2020), 2020, vol. 174, pp. 307—312.Valiev E., Yusupova N., Morozov A., Janschek K., Beyer M. Evaluation of the Impact of Random Computing Hardware Faults on the Performance of Convolutional Neural Networks, Proceedings of the 8th Scientific Conference on Information Technologies for Intelligent Decision Making Support (ITIDS 2020), 2020, vol. 174, pp. 307—312.Beyer M., Morozov A., Ding K., Ding S., and Janschek K. Quantification of the impact of random hardware faults on safetycritical ai applications: Cnn-based traffic sign recognition case study, Proceedings — 2019 IEEE 30th International Symposium on Software Reliability Engineering Workshops, ISSREW 2019, 2019, pp. 118—119.Beyer M., Morozov A., Ding K., Ding S., and Janschek K. Quantification of the impact of random hardware faults on safetycritical ai applications: Cnn-based traffic sign recognition case study, Proceedings — 2019 IEEE 30th International Symposium on Software Reliability Engineering Workshops, ISSREW 2019, 2019, pp. 118—119.Lowe R., Wu Y., Tamar A., Harb J., Abbeel P., Mordatch I. Multi-agent actor-critic for mixed cooperative-competitive environments, Advances in Neural Information Processing Systems, 2017, vol. 2017-December, pp. 1—12.Lowe R., Wu Y., Tamar A., Harb J., Abbeel P., Mordatch I. Multi-agent actor-critic for mixed cooperative-competitive environments, Advances in Neural Information Processing Systems, 2017, vol. 2017-December, pp. 1—12.Foerster J., Nardelli N., Farquhar G., Afouras T., Torr P., Kohli P., Shimon Whiteson S. Stabilising experience replay for deep multi-agent reinforcement learning, ICML’17: Proceedings of the 34th International Conference on Machine Learning, 2017, vol. 70, pp. 1146—1155.Foerster J., Nardelli N., Farquhar G., Afouras T., Torr P., Kohli P., Shimon Whiteson S. Stabilising experience replay for deep multi-agent reinforcement learning, ICML’17: Proceedings of the 34th International Conference on Machine Learning, 2017, vol. 70, pp. 1146—1155.Gupta J. K., Egorov M., Kochenderfer M. Cooperative Multi-agent Control Using Deep Reinforcement Learning, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2017, vol. 10642 LNAI, pp. 1—9.Gupta J. K., Egorov M., Kochenderfer M. Cooperative Multi-agent Control Using Deep Reinforcement Learning, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2017, vol. 10642 LNAI, pp. 1—9.Bloembergen D., Kaisers M., Tuyls K. Lenient frequency adjusted Q-learning, Belgian/Netherlands Artificial Intelligence Conference, 2010, pp. 19—26.Bloembergen D., Kaisers M., Tuyls K. Lenient frequency adjusted Q-learning, Belgian/Netherlands Artificial Intelligence Conference, 2010, pp. 19—26.Omidshafiei S., Pazis J., Amato C., How J. P., Vian J. Deep decentralized multi-task multi-agent reinforcement learning under partial observability, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, 2017, vol. 6, pp. 4108—4122.Omidshafiei S., Pazis J., Amato C., How J. P., Vian J. Deep decentralized multi-task multi-agent reinforcement learning under partial observability, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, 2017, vol. 6, pp. 4108—4122.Zheng Y., Jianye Hao J., Zhang Z. Weighted double deep multiagent reinforcement learning in stochastic cooperative environments, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018, vol. 11013 LNAI, pp. 1—8.Zheng Y., Jianye Hao J., Zhang Z. Weighted double deep multiagent reinforcement learning in stochastic cooperative environments, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018, vol. 11013 LNAI, pp. 1—8.Hong Z. W., Shih-Yang Su S. Y., Shann T. Y., Chang Y. H., Lee C. Y. A deep policy inference Q-network for multi-agent systems, Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2018, vol. 2, pp. 1388—1396.Hong Z. W., Shih-Yang Su S. Y., Shann T. Y., Chang Y. H., Lee C. Y. A deep policy inference Q-network for multi-agent systems, Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2018, vol. 2, pp. 1388—1396.Palmer G., Tuyls K., Bloembergen D., Savani R. Lenient multi-agent deep reinforcement learning, Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2018, vol. 1, pp. 1—9.Palmer G., Tuyls K., Bloembergen D., Savani R. Lenient multi-agent deep reinforcement learning, Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2018, vol. 1, pp. 1—9.Matignon L., Laurent G. J., Le Fort-Piat N. Hysteretic Q-Learning: An algorithm for decentralized reinforcement learning in cooperative multi-agent teams, IEEE International Conference on Intelligent Robots and Systems, 2007, pp. 1—7.Matignon L., Laurent G. J., Le Fort-Piat N. Hysteretic Q-Learning: An algorithm for decentralized reinforcement learning in cooperative multi-agent teams, IEEE International Conference on Intelligent Robots and Systems, 2007, pp. 1—7.Hausknecht M., Stone P. Deep recurrent q-learning for partially observable MDPs, AAAI Fall Symposium — Technical Report, 2015, pp. 29—37.Hausknecht M., Stone P. Deep recurrent q-learning for partially observable MDPs, AAAI Fall Symposium — Technical Report, 2015, pp. 29—37.Matignon L., Laurent G. J., Le Fort-Piat N. Independent reinforcement learners in cooperative Markov games: A survey regarding coordination problems, Knowledge Engineering Review, 2012, vol. 27, no. 1. pp. 1—32.Matignon L., Laurent G. J., Le Fort-Piat N. Independent reinforcement learners in cooperative Markov games: A survey regarding coordination problems, Knowledge Engineering Review, 2012, vol. 27, no. 1. pp. 1—32.Tan M. Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents, Machine Learning Proceedings 1993, 1993, pp. 1—8.Tan M. Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents, Machine Learning Proceedings 1993, 1993, pp. 1—8.Jaderberg M. et al. Human-level performance in 3D multiplayer games with population-based reinforcement learning, Science, 2019, vol. 364, no. 6443, pp. 859—865.Jaderberg M. et al. Human-level performance in 3D multiplayer games with population-based reinforcement learning, Science, 2019, vol. 364, no. 6443, pp. 859—865.Espeholt L. et al. IMPALA: Scalable Distributed DeepRL with Importance Weighted Actor-Learner Architectures, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, 2018, vol. 4, pp. 1—10.Espeholt L. et al. IMPALA: Scalable Distributed DeepRL with Importance Weighted Actor-Learner Architectures, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, 2018, vol. 4, pp. 1—10.Lillicrap T. P. et al. Continuous control with deep reinforcement learning, Proceedings of the 4th International Conference on Learning Representations, ICLR 2016 — Conference Track Proceedings, 2016, pp. 1—14.Lillicrap T. P. et al. Continuous control with deep reinforcement learning, Proceedings of the 4th International Conference on Learning Representations, ICLR 2016 — Conference Track Proceedings, 2016, pp. 1—14.Foerster J. N., Assael Y. M., Nando de Freitas N., Whiteson S. Learning to communicate with deep multi-agent reinforcement learning, Advances in Neural Information Processing Systems, 2016, pp. 1—9.Foerster J. N., Assael Y. M., Nando de Freitas N., Whiteson S. Learning to communicate with deep multi-agent reinforcement learning, Advances in Neural Information Processing Systems, 2016, pp. 1—9.Silver D., Lever G., Heess N., Degris T., Wierstra D., Riedmiller M. Deterministic policy gradient algorithms, Proceedings of the 31st International Conference on Machine Learning, ICML 2014, 2014, vol. 32, pp. 387—395.Silver D., Lever G., Heess N., Degris T., Wierstra D., Riedmiller M. Deterministic policy gradient algorithms, Proceedings of the 31st International Conference on Machine Learning, ICML 2014, 2014, vol. 32, pp. 387—395.

The authors declare that there are no conflicts of interest present.