Video Reflection Removal Through Spatio-Temporal Optimization

Ajay Nandoriya, Mohamed Elgharib, Changil Kim, Mohamed Hefeeda, Wojciech Matusik

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)

Abstract

Reflections can obstruct content during video capture and hence their removal is desirable. Current removal techniques are designed for still images, extracting only one reflection (foreground) and one background layer from the input. When extended to videos, unpleasant artifacts such as temporal flickering and incomplete separation are generated. We present a technique for video reflection removal by jointly solving for motion and separation. The novelty of our work is in our optimization formulation as well as the motion initialization strategy. We present a novel spatiotemporal optimization that takes n frames as input and directly estimates 2n frames as output, n for each layer. We aim to fully utilize spatio-temporal information in our objective terms. Our motion initialization is based on iterative frame-to-frame alignment instead of the direct alignment used by current approaches. We compare against advanced video extensions of the state of the art, and we significantly reduce temporal flickering and improve separation. In addition, we reduce image blur and recover moving objects more accurately. We validate our approach through subjective and objective evaluations on real and controlled data.

Original languageEnglish
Title of host publicationProceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages2430-2438
Number of pages9
Volume2017-October
ISBN (Electronic)9781538610329
DOIs
Publication statusPublished - 22 Dec 2017
Event16th IEEE International Conference on Computer Vision, ICCV 2017 - Venice, Italy
Duration: 22 Oct 201729 Oct 2017

Other

Other16th IEEE International Conference on Computer Vision, ICCV 2017
CountryItaly
CityVenice
Period22/10/1729/10/17

Fingerprint

Flickering

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition

Cite this

Nandoriya, A., Elgharib, M., Kim, C., Hefeeda, M., & Matusik, W. (2017). Video Reflection Removal Through Spatio-Temporal Optimization. In Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017 (Vol. 2017-October, pp. 2430-2438). [8237526] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCV.2017.264

Video Reflection Removal Through Spatio-Temporal Optimization. / Nandoriya, Ajay; Elgharib, Mohamed; Kim, Changil; Hefeeda, Mohamed; Matusik, Wojciech.

Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017. Vol. 2017-October Institute of Electrical and Electronics Engineers Inc., 2017. p. 2430-2438 8237526.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Nandoriya, A, Elgharib, M, Kim, C, Hefeeda, M & Matusik, W 2017, Video Reflection Removal Through Spatio-Temporal Optimization. in Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017. vol. 2017-October, 8237526, Institute of Electrical and Electronics Engineers Inc., pp. 2430-2438, 16th IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 22/10/17. https://doi.org/10.1109/ICCV.2017.264
Nandoriya A, Elgharib M, Kim C, Hefeeda M, Matusik W. Video Reflection Removal Through Spatio-Temporal Optimization. In Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017. Vol. 2017-October. Institute of Electrical and Electronics Engineers Inc. 2017. p. 2430-2438. 8237526 https://doi.org/10.1109/ICCV.2017.264
Nandoriya, Ajay ; Elgharib, Mohamed ; Kim, Changil ; Hefeeda, Mohamed ; Matusik, Wojciech. / Video Reflection Removal Through Spatio-Temporal Optimization. Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017. Vol. 2017-October Institute of Electrical and Electronics Engineers Inc., 2017. pp. 2430-2438
@inproceedings{8037748bf6c14962b470b20dafca9338,
title = "Video Reflection Removal Through Spatio-Temporal Optimization",
abstract = "Reflections can obstruct content during video capture and hence their removal is desirable. Current removal techniques are designed for still images, extracting only one reflection (foreground) and one background layer from the input. When extended to videos, unpleasant artifacts such as temporal flickering and incomplete separation are generated. We present a technique for video reflection removal by jointly solving for motion and separation. The novelty of our work is in our optimization formulation as well as the motion initialization strategy. We present a novel spatiotemporal optimization that takes n frames as input and directly estimates 2n frames as output, n for each layer. We aim to fully utilize spatio-temporal information in our objective terms. Our motion initialization is based on iterative frame-to-frame alignment instead of the direct alignment used by current approaches. We compare against advanced video extensions of the state of the art, and we significantly reduce temporal flickering and improve separation. In addition, we reduce image blur and recover moving objects more accurately. We validate our approach through subjective and objective evaluations on real and controlled data.",
author = "Ajay Nandoriya and Mohamed Elgharib and Changil Kim and Mohamed Hefeeda and Wojciech Matusik",
year = "2017",
month = "12",
day = "22",
doi = "10.1109/ICCV.2017.264",
language = "English",
volume = "2017-October",
pages = "2430--2438",
booktitle = "Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Video Reflection Removal Through Spatio-Temporal Optimization

AU - Nandoriya, Ajay

AU - Elgharib, Mohamed

AU - Kim, Changil

AU - Hefeeda, Mohamed

AU - Matusik, Wojciech

PY - 2017/12/22

Y1 - 2017/12/22

N2 - Reflections can obstruct content during video capture and hence their removal is desirable. Current removal techniques are designed for still images, extracting only one reflection (foreground) and one background layer from the input. When extended to videos, unpleasant artifacts such as temporal flickering and incomplete separation are generated. We present a technique for video reflection removal by jointly solving for motion and separation. The novelty of our work is in our optimization formulation as well as the motion initialization strategy. We present a novel spatiotemporal optimization that takes n frames as input and directly estimates 2n frames as output, n for each layer. We aim to fully utilize spatio-temporal information in our objective terms. Our motion initialization is based on iterative frame-to-frame alignment instead of the direct alignment used by current approaches. We compare against advanced video extensions of the state of the art, and we significantly reduce temporal flickering and improve separation. In addition, we reduce image blur and recover moving objects more accurately. We validate our approach through subjective and objective evaluations on real and controlled data.

AB - Reflections can obstruct content during video capture and hence their removal is desirable. Current removal techniques are designed for still images, extracting only one reflection (foreground) and one background layer from the input. When extended to videos, unpleasant artifacts such as temporal flickering and incomplete separation are generated. We present a technique for video reflection removal by jointly solving for motion and separation. The novelty of our work is in our optimization formulation as well as the motion initialization strategy. We present a novel spatiotemporal optimization that takes n frames as input and directly estimates 2n frames as output, n for each layer. We aim to fully utilize spatio-temporal information in our objective terms. Our motion initialization is based on iterative frame-to-frame alignment instead of the direct alignment used by current approaches. We compare against advanced video extensions of the state of the art, and we significantly reduce temporal flickering and improve separation. In addition, we reduce image blur and recover moving objects more accurately. We validate our approach through subjective and objective evaluations on real and controlled data.

UR - http://www.scopus.com/inward/record.url?scp=85041905220&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041905220&partnerID=8YFLogxK

U2 - 10.1109/ICCV.2017.264

DO - 10.1109/ICCV.2017.264

M3 - Conference contribution

VL - 2017-October

SP - 2430

EP - 2438

BT - Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -