Neural Network Method for Constructing Three-Dimensional Models of Rigid Objects from Satellite Images
https://doi.org/10.17587/mau.22.48-55
Abstract
A method has been developed for constructing three-dimensional models of rigid objects on the earth’s surface using one satellite image using the example of railway infrastructure. The method consists in step-by-step processing of satellite images with sequential application of two convolutional neural networks. In the first processing step, a satellite image is segmented by a neural network to select a plurality of objects of predetermined classes. At the second stage of processing with the help of neural network local analysis of image areas detected by results of the first stage of processing is performed. The results of the second processing step are used to estimate the parameters of the 3D model of the object. The possibilities of the method are shown by the example of processing a satellite image of the railway infrastructure. The following classes of informative areas are considered: building, wall edge, roof edge, building shadow, railway infrastructure, car, highway; rails, poles and shadows from poles (taken as reference objects for estimating scaling coefficients in certain directions). An example is given of using the developed method of highlighting typical railway infrastructure objects and for subsequent evaluation of the parameters of a three-dimensional building model partially obscured by trees.
Keywords
satellite images,
3D model,
raster area,
artificial neural network,
convolutional network,
machine learning,
infrastructure,
training sample
Supporting agencies: The research was carried out with the support of the Ministry of Science and Higher Education of the Russian Federation (unique project identifier RFMEFI60719X0312).
About the Authors
O. G. Gvozdev
Institute for Scientifi c Research of Aerospace Monitoring AEROCOSMOS; State University of Geodesy and Cartography
Russian Federation
Moscow, 105064
Russian Federation
Moscow, 105064
V. A. Kozub
Institute for Scientifi c Research of Aerospace Monitoring AEROCOSMOS
Russian Federation
Moscow, 105064
Russian Federation
Moscow, 105064
N. V. Kosheleva
Institute for Scientifi c Research of Aerospace Monitoring AEROCOSMOS
Russian Federation
Moscow, 105064
Russian Federation
Moscow, 105064
A. B. Murynin
Institute for Scientifi c Research of Aerospace Monitoring AEROCOSMOS; Federal Research Center "Computer Science and Control" of RAS
Russian Federation
Russian Federation
Ph.D., Leading Researcher
Moscow, 105064
Moscow, 119333
A. A. Richter
Institute for Scientifi c Research of Aerospace Monitoring AEROCOSMOS
Russian Federation
Russian Federation
PhD
Moscow, 105064
References
1. Tusikova E. M., Vikhtenko A. A. On road recognition on satellite imagery using convolutional mask-rcnn networks, V International Conference "Information Technologies and High Performance Computing" (ITHPC-2019), Khabarovsk, Russia, September 16—19, 2019, Proceedings, 2019, pp. 308—314 (in Russian).
2. Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, arXiv:1506.01497v3 [cs.CV] 6 Jan 2016, 56 p., available at: https://arxiv.org/abs/1506.01497.
3. Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick. Mask R-CNN, Facebook AI Research (FAIR), arXiv:1703.06870v3 [cs.CV] 24 Jan 2018, 10 p., available at: https://arxiv.org/abs/1703.06870.
4. Joseph Redmon, Ali Farhadi. YOLOv3: An Incremental Improvement / University of Washington, arXiv:1804.02767v1 [cs. CV] 8 Apr 2018, 6 p., available at: https://arxiv.org/abs/1804.02767.
5. Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg. SSD: Single Shot MultiBox Detector, arXiv:1512.02325v5 [cs.CV] 29 Dec 2016, doi 10.1007/978-3-319-46448-0_2, available at: https:// arxiv.org/abs/1512.02325.
6. Amaia Salvador, Míriam Bellver, Manel Baradad, Xavier Giró, Jordi Torres, Ferran Marqués. Recurrent Neural Networks for Semantic Instance Segmentation,Universitat Politècnica de Catalunya Barcelona Supercomputing Center, 2012, available at: https://imatge-upc.github.io/rsis.
7. Adrian Bulat, Jing Yang, Georgios Tzimiropoulos. To learn image super-resolution, use a GAN to learn how to do image degradation first, arXiv:1807.11458v1 [cs.CV], 30 Jul 2018, 16 p. available at: https://arxiv.org/abs/1807.11458.
8. Chen-Hsuan Lin, Chen Kong, Simon Lucey. Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction, The Robotics Institute, Carnegie Mellon University, arXiv:1706.07036v1 [cs.CV], 21 Jun 2017, 10 p., available at: https://arxiv.org/abs/1706.07036.
9. Davide Boscaini, Jonathan Masci, Emanuele Rodola`, Michael Bronstein. Learning shape correspondence with anisotropic convolutional neural networks, Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, 13 p., available at: https://arxiv.org/abs/1605.06437.
10. Li Yi, Hao Su, Xingwen Guo, Leonidas Guibas. SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation, arXiv:1612.00606v1 [cs.CV] 2 Dec 2016, 11 p., available at: https://arxiv.org/pdf/1612.00606.pdf.
11. Marte Otten, Yair Pinto, Chris L. E. Paffen, Anil K. Seth, Ryota Kanai. The Uniformity Illusion: Central Stimuli Can Determine Peripheral Perception, Psychological Science 2017, Vol. 28(1) 56—68, available at: https://journals.sagepub.com/doi/abs/10.1177/0956797616672270.
12. Wen Ma, Zongxu Pan, Feng Yuan, Bin Lei. Super-Resolution of Remote Sensing Images via a Dense Residual Generative Adversarial Network / Remote Sens. 2019, 11, 2578; doi:10.3390/ rs11212578, 24 p. available at: https://www.mdpi.com/2072-4292/11/21/2578/htm.
13. Fatemeh Alidoost, Hossein Arefi, Federico Tombari. 2D Image-To-3D Model: Knowledge-Based 3D Building Reconstruction (3DBR) Using Single Aerial Images and Convolutional Neural Networks (CNNs), Remote Sens., USA, 2019, vol.11, no. 2219, 25 p. doi:10.3390/rs11192219.
14. Fatemeh Alidoost, Hossein Arefi. A CNN-Based Approach for Automatic Building Detection and Recognition of Roof Types Using a Single Aerial Image, PFG, Germany, 2018, 86, pp. 235—248.
15. Park J. S. Interactive 3D reconstruction from multiple images: A primitivebased approach, Pattern Recognition Letters, 2005, no. 26(16), pp. 2558—2571.
16. Sherrie Wang, William Chen, Sang Michael Xie, George Azzari, David B. Lobell. Weakly Supervised Deep Learning for Segmentation of Remote Sensing Imagery, Remote Sens., USA, 2020, vol.12, no. 207, 25 p., doi:10.3390/rs12020207.
17. Arsalan Mousavian, Dragomir Anguelov John Flynn, Jana Košecká. 3D bounding box estimation using deep learning and geometry, arXiv:1612.00496v2 [cs.CV] 10 Apr 2017, 10 p. available at: https://arxiv.org/pdf/1612.00496.pdf.
18. Olaf Ronneberger, Philipp Fischer, Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation, Computer Vision and Pattern Recognition (cs.CV) 18 May 2015, available at: https://arxiv.org/abs/1505.04597.
19. Smith, S. M., Brady J. M. SUSAN — A New Approach to Low Level Image Processing, International Journal of Computer Vision, 1997, vol. 23, no. 1, pp. 45—78.
20. Trevor Standley, Ozan Sener, Dawn Chen, Silvio Savarese. Image2mass: estimating the massofan object from its image, 1st Conference on Robot Learning (CoRL 2017), Mountain View, United States, 10 p. available at: http://proceedings.mlr.press/v78/standley17a/standley17a.pdf.
21. Umut Gunes Sefercik, Serkan Karakis, Caglar Bayik, Mehmet Alkan, Naci Yastikli. Contribution of Normalized DSM to Automatic Building Extraction from HR Mono Optical Satellite Imagery, European Journal of Remote Sensing, Turkey, 2014, vol. 47, pp. 575—591. doi: 10.5721/EuJRS20144732.
22. Yaning Yi, Zhijie Zhang, Wanchang Zhang, Chuanrong Zhang, Weidong Li, Tian Zhao. Semantic segmentation of urban buildings from VHR remote sensing imagery using a deep convolutional neural network, Remote Sens., USA, 2019, vol. 11, no. 1774, 19 p., doi:10.3390/rs11151774.
23. Zhongyuan Wang, Kui Jiang, Peng Yi, Zhen Han, Zheng He. Ultra-dense GAN for satellite imagery super-resolution, Neurocomputing, 21 October 2019, available at: https://www.sciencedirect.com/science/article/pii/S0925231219314602, p. 328—337.
24. Bondur V. G. Aerospace methods and technologies for monitoring oil and gas territories and oil and gas complex facilities, Exploration of the Earth from space, 2010, no. 6, pp. 3—17 (in Russian).
25. Kazaryan M. L., Shakhramanyan M. A., Murynin A. B., Richter A. A. Construction of 3D models of rigid objects according to one image by the method of spatial grids, Information and Space, 2018, no. 4, pp. 90—102 (in Russian).
26. Murynin A. B., Richter A. A. Features of application of methods and algorithms of reconstruction of three-dimensional shape of rigid objects according to panoramic survey data, Machine Learning and Data Analysis, Russia, 2018. vol. 4, no. 4, pp. 235—247, doi: 10.21469/22233792.4.4.02 (in Russian).
27. Murynin A. B., Trekin A. N., Ignat’ev V. Yu., Kul’chenkova V. G., Rakova K. O. Method for increasing the resolution of space images of rigid objects, Machine Learning and Data Analysis, 2019, vol. 4, no. 5, pp. 296—308 (in Russian).
28. Sokolova N. O. Recognition of building contours on satellite images of high spatial resolution, Vestnik Kherson National Technical University, 2015, no. 3 (54), pp. 610—614 (in Russian).
29. Tuzhilkin A. Yu. Recognition and reconstruction of 3D objects from satellite images based on comparison of graph spectra, Fundamental research, 2015, no. 2, pp. 3727—3732 (in Russian).
30. Min Bai, Raquel Urtasun. Deep Watershed Transform for Instance Segmentation, Department of Computer Science, University of Toronto, arXiv:1611.08303v2 [cs.CV] 4 May 2017, 9 p., available at: https://arxiv.org/abs/1611.08303.
31. Kudryashov A. P., Soloviev I. V. Reconstruction of the textured model of urban space according to the topographic plan and photographs, Software products and systems, Software & Systems, Russia, 2019, no. 3 (32), doi: 10.15827/0236-235X.127.496-503 (in Russian).
32. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollar. Focal Loss for Dense Object Detection,arXiv:1708.02002v2 [cs.CV] 7 Feb 2018, available at: https://arxiv.org/pdf/1708.02002.pdf.
33. Richter A. A., Murynin A. B., Shlupikov V. A. Features of remote sensing data integration for the restoration of 3D models of rigid objects from satellite images, Materials of the International Scientific and Practical Conference "Achievements of Science in 2019", December 27, 2019, Center for Scientific Development "Big Book" (Moscow) (in Russian).
34. Gvozdev O. G., Murynin A. B., Richter A. A. Complex of applied solutions for the construction and training of artificial neural networks for semantic segmentation of aerospace images of an arbitrary channel-spectral structure in conditions of lack of training data, Materials of the 19th All-Russia. conf. with international participation: Mathematical methods of pattern recognition (ММРО-2019, Moscow), Russian Academy of Sciences, 2019, pp. 344—348 (in Russian).
35. Pytorch, available at: https://pytorch.org.
Review
For citations:
Gvozdev O.G., Kozub V.A., Kosheleva N.V., Murynin A.B., Richter A.A. Neural Network Method for Constructing Three-Dimensional Models of Rigid Objects from Satellite Images. Mekhatronika, Avtomatizatsiya, Upravlenie. 2021;22(1):48-55. (In Russ.) https://doi.org/10.17587/mau.22.48-55
Views:
769
.png)
Email this article 