Learned Variational Video Color Propagation

In this paper, we propose a novel method for color propagation that is used to recolor gray-scale videos (e.g. historic movies). Our energy-based model combines deep learning with a variational formulation. At its core, the method optimizes over a set of plausible color candidates that are extracted from motion and semantic feature matches, together with a learned regularizer that resolves color ambiguities by enforcing spatial smoothness.
Our approach allows to interpret intermediate results and to incorporate extensions like using multiple reference frames even after training.
We achieve state-of-the-art results on a number of standard benchmark datasets with multiple metrics and also provide convincing results on real historical videos - even though such types of video are not present during training.
Moreover, a user evaluation shows that our method propagates initial colors more faithfully and temporally consistent.