• Título:  MONOCULAR DEPTH ESTIMATION: DATASETS, METHODS, AND APPLICATIONS
• Autor:  Bauer, Zuria
• Universidad:  Universidad de Alicante
• Departamento:  Ciencia de la computación e inteligencia artificial
• Fecha de Lectura:  15/09/2021
• Programa de doctorado:  Programa de Doctorado en Informática por la Universidad de Alicante
• Dirección: 
  • CAZORLA QUEVEDO, MIGUEL ÁNGEL (DIRECTOR)
  • Orts Escolano, Sergio (CODIRECTOR)
• Tribunal: 
  • Martínez Martín, Ester (PRESIDENTE)
  • Gómez Donoso, Francisco Rafael (SECRETARIO)
  • Rangel Ortiz, Jose Carlos (VOCAL)
• Descriptores: 
  • INTELIGENCIA ARTIFICIAL
  • INFORMATICA
• Fichero de tesis:
• http://hdl.handle.net/10045/120297
• Resumen:
  1. Abstract The World Health Organization (WHO) stated in February 2021 at the Seventy-Third World Health Assembly that, globally, at least 2.2 billion people have a near or distance vision impairment. They also denoted the severe impact vision impairment has on the quality of life of the individual suffering from this condition, how it affects the social well-being and their economic independence in society, becoming in some cases an additional burden to also people in their immediate surroundings. In order to minimize the costs and intrusiveness of the applications and maximize the autonomy of the individual life, the natural solution is using systems that rely on computer vision algorithms. The systems improving the quality of life of the visually impaired need to solve different problems such as: localization, path recognition, obstacle detection, environment description, navigation, etc. Each of these topics involves an additional set of problems that have to be solved to address it. For example, for the task of object detection, there is the need of depth prediction to know the distance to the object, path recognition to know if the user is on the road or on a pedestrian path, alarm system to provide notifications of danger for the user, trajectory prediction of the approaching obstacle, and those are only the main key points. Taking a closer look at all of these topics, they have one key component in common: depth estimation/prediction. All of these topics are in need of a correct estimation of the depth in the scenario. In this thesis, our main focus relies on addressing depth estimation in indoor and outdoor environments. Traditional depth estimation methods, like structure from motion and stereo matching, are built on feature correspondences from multiple viewpoints. Despite the effectiveness of these approaches, they need a specific type of data for their proper performance. Since our main goal is to provide systems with minimal costs and intrusiveness that are also easy to handle we decided to infer the depth from single images: monocular depth estimation.  Estimating depth of a scene from a single image is a simple task for humans, but it is notoriously more difficult for computational models to be able to achieve high accuracy and low resource requirements. Monocular Depth Estimation is this very task of estimating depth from a single RGB image. Since there is only a need of one image, this approach is used in applications such as autonomous driving, scene understanding or 3D modeling where other type of information is not available. This thesis presents contributions towards solving this task using deep learning as the main tool. The four main contributions of this thesis are: first, we carry out an extensive review of the state-of-the-art in monocular depth estimation; secondly, we introduce a novel large scale high resolution outdoor stereo dataset able to provide enough image information to solve various common computer vision problems; thirdly, we show a set of architectures able to predict monocular depth effectively; and, at last, we propose two real life applications of those architectures, addressing the topic of enhancing the perception for the visually impaired using low-cost wearable sensors. 2. Introduction In this doctoral thesis, theoretical and practical research was conducted to explore the challenge of perceiving the environment, i.e., by providing enough information to a system to perceive and act based on it. Since this issue includes a lot of different application areas, the main focus of this work is the estimation of a depth map from monocular frames, to gain a better understanding of the geometry of the scene. Novel DL methods able to predict the distance relative to a camera are studied in outdoor and indoor environments. Moreover, some of those methods are used in applications developed to address real life problems. 3. Motivation The concept of AI has been in our world for several years now. It is impossible to ignore the potential of machine learning in the right spaces. When properly utilized, AI can transform the world, making it easier to process huge amounts of data and make important decisions.  Measuring distance relative to a camera remains difficult but absolutely key to unlocking exciting applications such as autonomous driving, three-dimensional (3D) scene reconstruction, 3D mapping and AR. Humans have been trying to estimate different and more accurate ways to find distance to objects whether it may be applied to robotics for obstacle avoidance, perception or navigation. Depth estimation from monocular frames is a blooming field in computer vision. Being able to perform accurate depth map predictions carries evident advantages, as any color camera would sense tridimensional data. Furthermore, using 2 or more cameras to triangulate the depth of the scene, although it increases the accuracy, it also adds more complexity, i.e. the cameras have to be constantly recalibrated due to the movement of the autonomous car or robot. Continuing on the same topic but slightly changing the viewpoint, we are going to focus on a more social approach. According to the Prevention of Blindness and Deafness Program of WHO, about 285 million people are visually impaired worldwide. Thirty-nine million are blind, and 246 million have low vision (severe or moderate visual impairment). Some of the key features an \ac{AT} should have, are the ability to predict depth of the scene, detect the objects in it and give warnings to help the user to avoid obstacles. Fusing monocular depth estimation with the idea of creating an \ac{AT} for visually impaired implies the ability to create low-cost devices for alerting those individuals of obstacles, generating tridimensional reconstructions of an object with no dedicated devices, or for instance, implementing it in smaller assistive mobile robots that require less hardware. 4. Conclusions 4.1. General Conclusions In this thesis, we have explored the field of monocular depth prediction in computer vision. First, after doing a review of the state-of-the-art of the different topics addressed in this thesis, it can be stated that DL architectures outperform traditional hand-engineered approaches. We proved that \ac{DL} methods for depth estimation from monocular RGB images, without temporal or multi-view information, provide accurate depth predictions, as well as proven useful in real-world applications. During this thesis we proved to be able to train three different networks and use them on real world data. We also proved the effectiveness of those networks as part of bigger application pipelines thus proving the capability of these architectures to estimate accurate depth in real life scenarios. Across all the architectures studied we experienced difficulties in training the models due to different problems such as: memory usage to train the network or overfitting of the networks on the training data, as well as reproducibility issues of the networks. We were able to address some of the problems, like the overfitting one, by introducing our own dataset providing enough data and variety of data to ensure generalization abilities of the different architectures. Still for some architectures the provided data can not be enough, proving that there still is work that needs to be performed on this topic. 4.2 Contributions of the Thesis Once we have stated the general conclusions of the thesis, we briefly summarize the main contributions and findings. Chapter 2 introduced a novel large scale high resolution outdoor stereo dataset for various tasks such as depth estimation (stereo or monocular), geolocalization, 3D reconstruction or object detection. It also addressed the problem with the amount of provided images, being suitable to train DL architectures without the risk of overfitting. It also is one of the few datasets that was filmed outdoors and features human interaction. Chapter 3 studied, analyzed and evaluated one of the state-of-the-art methods based on a fully convolutional model introduced by [Laina et al.,2016]. The network proved good generalization capabilities in predicting depth of unknown environments that were not used during training. It also proved the capability of handling correctly optical artifacts in the scene compared to the pretrained model, e.g: while handling sun light rays in the scene. Furthermore, the network was evaluated on an indoor real, indoor synthetic and outdoor real datasets to provide a qualitative comparison between the pretrained model and the newly implemented one proving its capability. In Chapter 4, we proposed a lightweight GAN architecture able to perform in indoor and outdoor environments, proving that the scale is not a decisive factor. Additionally, the network showed its ability to yield good predictions on difficult scenarios such as low textured areas. Furthermore, the network proved to be competitive enough with a 50 layer network such as ResNet-50. Chapter 5 proposed a novel pipeline for monocular depth prediction taking advantage of novel view synthesis. We presented a novel pipeline for monocular depth estimation. The pipeline leads to a decisive improvement in the state-of-art for the KITTI dataset, being able to outperform partially the baseline architectures. Additionally we provided results on the synthetic dataset Replica, showing also there the performance of the pipeline. Overall, we proposed a effective pipeline approach to improve monocular depth estimation architectures providing additional consistency during training. Chapter 6 proposed a system to enhance the perception capabilities of the visually impaired. The system generated a three-dimensional representation of the scenes with semantic labels for the obstacles from the feed of a wearable camera. Two wrist bands provided the haptic feedback in order to communicate to the user. Chapter 7 proposed two methods to improve the quality of the Pepper 1.8a depth map. For the first method we took advantages of monocular depth estimation fused with the point cloud produced by the Pepper, the planar areas were found being able to generate a corrected point cloud. For the second method we presented a refinement of the previously proposed one. This was handled by making a fusion between the Pepper and monocular point cloud, aligning the second with the first one and looking for planar areas. Thus, we provided a point cloud of better quality, preserving the density of points of the monocular depth estimation. 6. Bibliography [Abadi et al., 2015] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. (2015). TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. 90 [Aleotti et al., 2019] Aleotti, F., Tosi, F., Poggi, M., and Mattoccia, S. (2019). Generative adversarial networks for unsupervised monocular depth prediction. In Leal-Taixé, L. and Roth, S., editors, Computer Vision ¿ ECCV 2018 Workshops, pages 337¿354, Cham. Springer International Publishing. 63 [Almalioglu et al., 2018] Almalioglu, Y., Saputra, M. R. U., de Gusmao, P. P. B., Markham, A., and Trigoni, N. (2018). GANVO: unsupervised deep monocular visual odometry and depth estimation with generative adversarial networks. CoRR, abs/1809.05786. 11, 64, 196 [Andersen et al., 2012] Andersen, M., Jensen, T., Lisouski, P., Mortensen, A., Hansen, M., Gregersen, T., and Ahrendt, P. (2012). Kinect Depth Sensor Evaluation for Computer Vision Applications. Aarhus University, Department of Engineering. 137 [Armeni et al., 2017] Armeni, I., Sax, S., Zamir, A. R., and Savarese, S. (2017). Joint 2d-3d-semantic data for indoor scene understanding. CoRR, abs/1702.01105. 21, 91 [Bauer et al., 2020] Bauer, Z., Dominguez, A., Cruz, E., Gomez-Donoso, F., Orts-Escolano, S., and Cazorla, M. (2020). Enhancing perception for the visually impaired with deep learning techniques and low-cost wearable sensors. Pattern recognition letters, 137:27¿36. 166, 206 [Bauer et al., 2018] Bauer, Z., Escalona, F., Cruz, E., Cazorla, M., and Gomez-Donoso, F. (2018). Improving the 3d perception of the pepper robot using depth prediction from monocular frames. In Workshop of Physical Agents, pages 132¿146. Springer. 166, 206 [Bauer et al., 2019a] Bauer, Z., Escalona, F., Cruz, E., Cazorla, M., and Gomez-Donoso, F. (2019a). Refining the fusion of pepper robot and estimated depth maps method for improved 3d perception. IEEE Access, 7:185076¿185085. 166, 206 [Bauer et al., 2019b] Bauer, Z., Gomez-Donoso, F., Cruz, E., Orts-Escolano, S., and Cazorla, M. (2019b). Uasol, a large-scale high-resolution outdoor stereo dataset. Scientific Data, 6. xxx, 21, 22, 35, 91, 92, 122, 166, 206 [Bhat et al., 2020] Bhat, S. F., Alhashim, I., and Wonka, P. (2020). Adabins: Depth estimation using adaptive bins. xxv, 8, 95, 96, 97, 192 [Bolme et al., 2010] Bolme, D. S., Beveridge, J. R., Draper, B. A., and Lui, Y. M. (2010). Visual object tracking using adaptive correlation filters. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 2544¿2550. 118 [Cabrera et al., 2018] Cabrera, E. V., Ortiz, L. E., Silva, B. M. F. d., Clua, E. W. G., and Gonçalves, L. M. G. (2018). A versatile method for depth data error estimation in rgb-d sensors. Sensors (Basel, Switzerland), 18(9). 28 (due to space limitations only a part of the bibliography is included here)