Fine-tuning Monocular Depth-Estimator Artificial Neural Networks Trained on Synthetic RGB-D Data Sets for Real Scenes

With every new synthetic RGB-D data set, the scenes are becoming more and more photo-realistic. However, the depth-estimator models trained on these sophisticated data sets still fail in generalization to real scenes. Recent attempts to overcome this challenge mainly include transferring style of the RGB images from synthetic to real or vice versa. Such approaches make it possible to train the depth-estimator without the availability of real image-depth pairs. On the other hand, this study considers a case in which a limited number of real RGB-D samples are also available. The real samples' availability allows for fine-tuning as an approach to fill the gap between the style of the real and synthetic samples. Results show that no matter how many real samples are available, to fit the real samples to all the parameters of the pre-trained model (considered for this study) is always better than only fine-tuning the parameters of the encoder of the model. Also, it is in the availability of less than 12.5% of the real data required to achieve the best performance of the model that complete fine-tuning provides results better than training from scratch. This study, also details the process of building a synthetic data for flaying robots in the Unity game Engine.