ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2015)
 
Blind Video Temporal Consistency

Nicolas Bonneel James Tompkin Kalyan Sunkavalli Deqing Sun Sylvain Paris Hanspeter Pfister
LIRIS - CNRS Harvard University SEAS Adobe

(a) Input video. (b) Per-frame processing (c) Our result
With many image filters, such as this automatic color, tone, and contrast adjustment, processing an input video (a) (frames 167-168) frame by frame results in temporal discontinuities (b). We take the two video sequences (a) and (b) and automatically generate a temporally consistent video (c), without knowing the image filter used to produce the unstable video (b). This enables the application of our technique to a wide range of video effects such as color constancy, stylization, color grading, intrinsic decomposition, depth prediction, and dehazing.


Abstract
Extending image processing techniques to videos is a non-trivial task; applying processing independently to each video frame often leads to temporal inconsistencies, and explicitly encoding temporal consistency requires algorithmic changes. We describe a more general approach to temporal consistency. We propose a gradientdomain technique that is blind to the particular image processing algorithm. Our technique takes a series of processed frames that suffers from flickering and generates a temporally-consistent video sequence. The core of our solution is to infer the temporal regularity from the original unprocessed video, and use it as a temporal consistency guide to stabilize the processed sequence. We formally characterize the frequency properties of our technique, and demonstrate, in practice, its ability to stabilize a wide range of popular image processing techniques including enhancement and stylization of color and tone, intrinsic images, and depth estimation.

 
@article{BTSSPP15,
author = {Nicolas Bonneel and James Tompkin and Kalyan Sunkavalli
and Deqing Sun and Sylvain Paris and Hanspeter Pfister},
title = {Blind Video Temporal Consistency},
journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2015)},
volume = {34},
number = {6},
year = {2015},
}

   
Paper
PDF (10 MB)
  Paper (low res.)
PDF (1 MB)
  Presentation
PPTX (150 MB)
  Supplemental Material
ZIP (1.8 GB)
  Code and binary
GIT


 
  Supplemental Video
MP4 (93 MB) | YouTube


Acknowledgements
We thank Eugene Hsu and Ivaylo Boyadzhiev for their white balance implementation [Hsu et al. 2008], Xue Bai and Kang In Kim for their suggestions, Pierre Jasmin of RE:Vision for his expertise in processing sequences with DE:Flicker [RE:Vision 2015], and Adobe for their donation.
We thank the authors of the video footage: G. Mougin (Fig. 1), Unbound (Fig. 4), Millgate (Fig. 5 and 9), G. Henkel (Fig. 6), S. Pyeatte (Zina Nicole Lahr, Fig. 7 and 9), B. Fitzgerald (Fig. 9), P. Shin (Fig. 9) and B. Bourgon (Fig. 8). Used with permission or under CC BY-NA licence.
Copyright by the authors, 2015. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in ACM Transactions on Graphics: http://dx.doi.org/10.1145/2661229.2661253
Zip icon adapted from tastic mimetypes by Untergunter, CC BY-NA-SA 3.0 licence.
This work was partially supported by NSF grants NSF CGV-1110955, with the work also sponsored by the Air Force Research Laboratory and DARPA Memex program.