Multicamera audio-visual analysis of dance figures

Ferda Ofli, Y. Demir, E. Erzin, Y. Yemez, A. M. Tekalp

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Citations (Scopus)

Abstract

We present an automated system for multicamera motion capture and audio-visual analysis of dance figures. The multiview video of a dancing actor is acquired using 8 synchronized cameras. The motion capture technique is based on 3D tracking of the markers attached to the person's body in the scene, using stereo color information without need for an explicit 3D model. The resulting set of 3D points is then used to extract the body motion features as 3D displacement vectors whereas MFC coefficients serve as the audio features. In the first stage of multimodal analysis, we perform Hidden Markov Model (HMM) based unsupervised temporal segmentation of the audio and body motion features, separately, to determine the recurrent elementary audio and body motion patterns. Then in the second stage, we investigate the correlation of body motion patterns with audio patterns, that can be used for estimation and synthesis of realistic audio-driven body animation.

Original languageEnglish
Title of host publicationProceedings of the 2007 IEEE International Conference on Multimedia and Expo, ICME 2007
Pages1703-1706
Number of pages4
Publication statusPublished - 1 Dec 2007
Externally publishedYes
EventIEEE International Conference onMultimedia and Expo, ICME 2007 - Beijing, China
Duration: 2 Jul 20075 Jul 2007

Other

OtherIEEE International Conference onMultimedia and Expo, ICME 2007
CountryChina
CityBeijing
Period2/7/075/7/07

Fingerprint

Hidden Markov models
Animation
Cameras
Color

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Software

Cite this

Ofli, F., Demir, Y., Erzin, E., Yemez, Y., & Tekalp, A. M. (2007). Multicamera audio-visual analysis of dance figures. In Proceedings of the 2007 IEEE International Conference on Multimedia and Expo, ICME 2007 (pp. 1703-1706). [4284997]

Multicamera audio-visual analysis of dance figures. / Ofli, Ferda; Demir, Y.; Erzin, E.; Yemez, Y.; Tekalp, A. M.

Proceedings of the 2007 IEEE International Conference on Multimedia and Expo, ICME 2007. 2007. p. 1703-1706 4284997.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Ofli, F, Demir, Y, Erzin, E, Yemez, Y & Tekalp, AM 2007, Multicamera audio-visual analysis of dance figures. in Proceedings of the 2007 IEEE International Conference on Multimedia and Expo, ICME 2007., 4284997, pp. 1703-1706, IEEE International Conference onMultimedia and Expo, ICME 2007, Beijing, China, 2/7/07.
Ofli F, Demir Y, Erzin E, Yemez Y, Tekalp AM. Multicamera audio-visual analysis of dance figures. In Proceedings of the 2007 IEEE International Conference on Multimedia and Expo, ICME 2007. 2007. p. 1703-1706. 4284997
Ofli, Ferda ; Demir, Y. ; Erzin, E. ; Yemez, Y. ; Tekalp, A. M. / Multicamera audio-visual analysis of dance figures. Proceedings of the 2007 IEEE International Conference on Multimedia and Expo, ICME 2007. 2007. pp. 1703-1706
@inproceedings{a80339d9a53549e3b881f91b68fbe9e0,
title = "Multicamera audio-visual analysis of dance figures",
abstract = "We present an automated system for multicamera motion capture and audio-visual analysis of dance figures. The multiview video of a dancing actor is acquired using 8 synchronized cameras. The motion capture technique is based on 3D tracking of the markers attached to the person's body in the scene, using stereo color information without need for an explicit 3D model. The resulting set of 3D points is then used to extract the body motion features as 3D displacement vectors whereas MFC coefficients serve as the audio features. In the first stage of multimodal analysis, we perform Hidden Markov Model (HMM) based unsupervised temporal segmentation of the audio and body motion features, separately, to determine the recurrent elementary audio and body motion patterns. Then in the second stage, we investigate the correlation of body motion patterns with audio patterns, that can be used for estimation and synthesis of realistic audio-driven body animation.",
author = "Ferda Ofli and Y. Demir and E. Erzin and Y. Yemez and Tekalp, {A. M.}",
year = "2007",
month = "12",
day = "1",
language = "English",
isbn = "1424410177",
pages = "1703--1706",
booktitle = "Proceedings of the 2007 IEEE International Conference on Multimedia and Expo, ICME 2007",

}

TY - GEN

T1 - Multicamera audio-visual analysis of dance figures

AU - Ofli, Ferda

AU - Demir, Y.

AU - Erzin, E.

AU - Yemez, Y.

AU - Tekalp, A. M.

PY - 2007/12/1

Y1 - 2007/12/1

N2 - We present an automated system for multicamera motion capture and audio-visual analysis of dance figures. The multiview video of a dancing actor is acquired using 8 synchronized cameras. The motion capture technique is based on 3D tracking of the markers attached to the person's body in the scene, using stereo color information without need for an explicit 3D model. The resulting set of 3D points is then used to extract the body motion features as 3D displacement vectors whereas MFC coefficients serve as the audio features. In the first stage of multimodal analysis, we perform Hidden Markov Model (HMM) based unsupervised temporal segmentation of the audio and body motion features, separately, to determine the recurrent elementary audio and body motion patterns. Then in the second stage, we investigate the correlation of body motion patterns with audio patterns, that can be used for estimation and synthesis of realistic audio-driven body animation.

AB - We present an automated system for multicamera motion capture and audio-visual analysis of dance figures. The multiview video of a dancing actor is acquired using 8 synchronized cameras. The motion capture technique is based on 3D tracking of the markers attached to the person's body in the scene, using stereo color information without need for an explicit 3D model. The resulting set of 3D points is then used to extract the body motion features as 3D displacement vectors whereas MFC coefficients serve as the audio features. In the first stage of multimodal analysis, we perform Hidden Markov Model (HMM) based unsupervised temporal segmentation of the audio and body motion features, separately, to determine the recurrent elementary audio and body motion patterns. Then in the second stage, we investigate the correlation of body motion patterns with audio patterns, that can be used for estimation and synthesis of realistic audio-driven body animation.

UR - http://www.scopus.com/inward/record.url?scp=46449111503&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=46449111503&partnerID=8YFLogxK

M3 - Conference contribution

SN - 1424410177

SN - 9781424410170

SP - 1703

EP - 1706

BT - Proceedings of the 2007 IEEE International Conference on Multimedia and Expo, ICME 2007

ER -