End-to-End Training of Hybrid CNN-CRF Models for Semantic Segmentation using Structured Learning

Aleksander Colovic, Patrick Knöbelreiter, Alexander Shekhovtsov, Thomas Pock

Research output: Contribution to conferencePaperpeer-review


In this work we tackle the problem of semantic image segmentation with a combination of convolutional neural networks (CNNs) and conditional random fields (CRFs). The CRF takes contrast sensitive weights in a local neighborhood as input (pairwise interactions) to encourage consistency (smoothness) within the prediction and align our segmentation boundaries with visual edges. We model unary terms with a CNN which outperforms non data driven models. We approximate the CRF inference with a fixed number of iterations of a linear-programming relaxation based approach. We experiment with training the combined model end-to-end using a discriminative formulation (structured support vector machine) and applying stochastic subgradient descend to it.
Our proposed model achieves an intersection over union score of 62.4 in the test set of the cityscapes pixel-level semantic labeling task which is comparable to state-of-the-art models.
Original languageEnglish
Publication statusPublished - 6 Feb 2017
EventComputer Vision Winter Workshop: CVWW 2017 - Retz, Retz, Austria
Duration: 6 Feb 20178 Feb 2017


ConferenceComputer Vision Winter Workshop
Abbreviated titleCVWW 2017

Cite this