Comparison of monocular depth estimation methods using geometrically relevant metrics on the IBims-1 dataset

The task of predicting a dense depth map from a monocular RGB image, commonly known as single-image depth estimation (SIDE) or monocular depth estimation (MDE), is an active research topic in computer vision for decades. With the significant progress of deep models in recent years, new standards were set yielding remarkable results in capturing the 3D structure from a single image. However, established evaluation schemes of predicted depth maps are still limited, as they only consider global statistics of the depth residuals. In order to allow for a geometry-aware analysis, we propose a set of novel quality criteria addressing the preservation of depth discontinuities and planar regions, the depth consistency across the image, and a distance-related assessment. As current datasets do not fulfill the requirements of all proposed error metrics, we provide a new high-quality indoor RGB-D test dataset, acquired by a digital single-lens reflex (DSLR) camera together with a laser scanner. New insights into the performance of current state-of-the-art SIDE approaches, as well as subtle differences among them, could be unveiled by employing the proposed error metrics on our reference dataset. Additionally, investigations on the real-world applicability of SIDE methods by a series of experiments regarding different image augmentations, illumination changes and textured planar regions have shown current limitations in this research field.