Data Driven 2-D-To-3-D Video Conversion for Soccer

Kiana Calagari, Mohamed Elgharib, Piotr Didyk, Alexandre Kaspar, Wojciech Matusik, Mohamed Hefeeda

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

A wide adoption of 3-D videos is hindered by the lack of high-quality 3-D content. One promising solution to this problem is through data-driven 2-D-To-3-D video conversion. Such approaches are based on learning depth maps from a large dataset of 2-D+Depth images. However, current conversion methods, while general, produce low-quality results with artifacts that are not acceptable to many viewers. We propose a novel, data-driven method for 2-D-To-3-D video conversion. Our method transfers the depth gradients from a large database of 2-D+Depth images. Capturing 2-D+Depth databases, however, are complex and costly, especially for outdoor sports games. We address this problem by creating a synthetic database from computer games and showing that this synthetic database can effectively be used to convert real videos. We propose a spatio-Temporal method to ensure the smoothness of the generated depth within individual frames and across successive frames. In addition, we present an object boundary detection method customized for 2-D-To-3-D conversion systems, which produces clear depth boundaries for players. We implement our method and validate it by conducting user studies that evaluate depth perception and visual comfort of the converted 3-D videos. We show that our method produces high-quality 3-D videos that are almost indistinguishable from videos shot by stereo cameras. In addition, our method significantly outperforms the current state-of-The-Art methods. For example, up to 20% improvement in the perceived depth is achieved by our method, which translates to improving the mean opinion score from good to excellent.

Original languageEnglish
Pages (from-to)605-619
Number of pages15
JournalIEEE Transactions on Multimedia
Volume20
Issue number3
DOIs
Publication statusPublished - 1 Mar 2018

Fingerprint

Depth perception
Computer games
Sports
Cameras

Keywords

  • 2-D-To-3-D conversion
  • depth estimation
  • three-dimensional (3-D) video

ASJC Scopus subject areas

  • Signal Processing
  • Media Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering

Cite this

Calagari, K., Elgharib, M., Didyk, P., Kaspar, A., Matusik, W., & Hefeeda, M. (2018). Data Driven 2-D-To-3-D Video Conversion for Soccer. IEEE Transactions on Multimedia, 20(3), 605-619. https://doi.org/10.1109/TMM.2017.2748458

Data Driven 2-D-To-3-D Video Conversion for Soccer. / Calagari, Kiana; Elgharib, Mohamed; Didyk, Piotr; Kaspar, Alexandre; Matusik, Wojciech; Hefeeda, Mohamed.

In: IEEE Transactions on Multimedia, Vol. 20, No. 3, 01.03.2018, p. 605-619.

Research output: Contribution to journalArticle

Calagari, K, Elgharib, M, Didyk, P, Kaspar, A, Matusik, W & Hefeeda, M 2018, 'Data Driven 2-D-To-3-D Video Conversion for Soccer', IEEE Transactions on Multimedia, vol. 20, no. 3, pp. 605-619. https://doi.org/10.1109/TMM.2017.2748458
Calagari, Kiana ; Elgharib, Mohamed ; Didyk, Piotr ; Kaspar, Alexandre ; Matusik, Wojciech ; Hefeeda, Mohamed. / Data Driven 2-D-To-3-D Video Conversion for Soccer. In: IEEE Transactions on Multimedia. 2018 ; Vol. 20, No. 3. pp. 605-619.
@article{65c8d222ebd746b18977b7291697c46a,
title = "Data Driven 2-D-To-3-D Video Conversion for Soccer",
abstract = "A wide adoption of 3-D videos is hindered by the lack of high-quality 3-D content. One promising solution to this problem is through data-driven 2-D-To-3-D video conversion. Such approaches are based on learning depth maps from a large dataset of 2-D+Depth images. However, current conversion methods, while general, produce low-quality results with artifacts that are not acceptable to many viewers. We propose a novel, data-driven method for 2-D-To-3-D video conversion. Our method transfers the depth gradients from a large database of 2-D+Depth images. Capturing 2-D+Depth databases, however, are complex and costly, especially for outdoor sports games. We address this problem by creating a synthetic database from computer games and showing that this synthetic database can effectively be used to convert real videos. We propose a spatio-Temporal method to ensure the smoothness of the generated depth within individual frames and across successive frames. In addition, we present an object boundary detection method customized for 2-D-To-3-D conversion systems, which produces clear depth boundaries for players. We implement our method and validate it by conducting user studies that evaluate depth perception and visual comfort of the converted 3-D videos. We show that our method produces high-quality 3-D videos that are almost indistinguishable from videos shot by stereo cameras. In addition, our method significantly outperforms the current state-of-The-Art methods. For example, up to 20{\%} improvement in the perceived depth is achieved by our method, which translates to improving the mean opinion score from good to excellent.",
keywords = "2-D-To-3-D conversion, depth estimation, three-dimensional (3-D) video",
author = "Kiana Calagari and Mohamed Elgharib and Piotr Didyk and Alexandre Kaspar and Wojciech Matusik and Mohamed Hefeeda",
year = "2018",
month = "3",
day = "1",
doi = "10.1109/TMM.2017.2748458",
language = "English",
volume = "20",
pages = "605--619",
journal = "IEEE Transactions on Multimedia",
issn = "1520-9210",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "3",

}

TY - JOUR

T1 - Data Driven 2-D-To-3-D Video Conversion for Soccer

AU - Calagari, Kiana

AU - Elgharib, Mohamed

AU - Didyk, Piotr

AU - Kaspar, Alexandre

AU - Matusik, Wojciech

AU - Hefeeda, Mohamed

PY - 2018/3/1

Y1 - 2018/3/1

N2 - A wide adoption of 3-D videos is hindered by the lack of high-quality 3-D content. One promising solution to this problem is through data-driven 2-D-To-3-D video conversion. Such approaches are based on learning depth maps from a large dataset of 2-D+Depth images. However, current conversion methods, while general, produce low-quality results with artifacts that are not acceptable to many viewers. We propose a novel, data-driven method for 2-D-To-3-D video conversion. Our method transfers the depth gradients from a large database of 2-D+Depth images. Capturing 2-D+Depth databases, however, are complex and costly, especially for outdoor sports games. We address this problem by creating a synthetic database from computer games and showing that this synthetic database can effectively be used to convert real videos. We propose a spatio-Temporal method to ensure the smoothness of the generated depth within individual frames and across successive frames. In addition, we present an object boundary detection method customized for 2-D-To-3-D conversion systems, which produces clear depth boundaries for players. We implement our method and validate it by conducting user studies that evaluate depth perception and visual comfort of the converted 3-D videos. We show that our method produces high-quality 3-D videos that are almost indistinguishable from videos shot by stereo cameras. In addition, our method significantly outperforms the current state-of-The-Art methods. For example, up to 20% improvement in the perceived depth is achieved by our method, which translates to improving the mean opinion score from good to excellent.

AB - A wide adoption of 3-D videos is hindered by the lack of high-quality 3-D content. One promising solution to this problem is through data-driven 2-D-To-3-D video conversion. Such approaches are based on learning depth maps from a large dataset of 2-D+Depth images. However, current conversion methods, while general, produce low-quality results with artifacts that are not acceptable to many viewers. We propose a novel, data-driven method for 2-D-To-3-D video conversion. Our method transfers the depth gradients from a large database of 2-D+Depth images. Capturing 2-D+Depth databases, however, are complex and costly, especially for outdoor sports games. We address this problem by creating a synthetic database from computer games and showing that this synthetic database can effectively be used to convert real videos. We propose a spatio-Temporal method to ensure the smoothness of the generated depth within individual frames and across successive frames. In addition, we present an object boundary detection method customized for 2-D-To-3-D conversion systems, which produces clear depth boundaries for players. We implement our method and validate it by conducting user studies that evaluate depth perception and visual comfort of the converted 3-D videos. We show that our method produces high-quality 3-D videos that are almost indistinguishable from videos shot by stereo cameras. In addition, our method significantly outperforms the current state-of-The-Art methods. For example, up to 20% improvement in the perceived depth is achieved by our method, which translates to improving the mean opinion score from good to excellent.

KW - 2-D-To-3-D conversion

KW - depth estimation

KW - three-dimensional (3-D) video

UR - http://www.scopus.com/inward/record.url?scp=85029158223&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85029158223&partnerID=8YFLogxK

U2 - 10.1109/TMM.2017.2748458

DO - 10.1109/TMM.2017.2748458

M3 - Article

VL - 20

SP - 605

EP - 619

JO - IEEE Transactions on Multimedia

JF - IEEE Transactions on Multimedia

SN - 1520-9210

IS - 3

ER -