Efficient Multi-Task Learning of Semantic Segmentation and Disparity Estimation

We propose a jointly trainable model for semantic
segmentation and disparity map estimation. In this work we
utilize the fact that the two tasks have complementary strength
and weaknesses. Traditional depth prediction algorithms rely on
low-level features and often have problems at large textureless
regions, while for semantic segmentation these regions are
easier to capture. We propose a CNN-based architecture, where
both tasks are tightly interconnected to each other. The model
consists of an encoding stage which computes features for both
tasks, semantic segmentation and disparity estimation. In the
decoding stage we explicitly add the semantic predictions to
the disparity decoding branch and we additionally allow to exchange information in the intermediate feature representations.
Furthermore, we set the focus on efficiency, which we achieve
by the usage of previously introduced ESP building blocks. We
evaluate the model on the commonly used KITTI dataset.