Sequence of the Most Informative Joints (SMIJ): A new representation for human skeletal action recognition

Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, René Vidal, Ruzena Bajcsy

Research output: Chapter in Book/Report/Conference proceedingConference contribution

52 Citations (Scopus)

Abstract

Much of the existing work on action recognition combines simple features (e.g., joint angle trajectories, optical flow, spatio-temporal video features) with somewhat complex classifiers or dynamical models (e.g., kernel SVMs, HMMs, LDSs, deep belief networks). Although successful, these approaches represent an action with a set of parameters that usually do not have any physical meaning. As a consequence, such approaches do not provide any qualitative insight that relates an action to the actual motion of the body or its parts. For example, it is not necessarily the case that clapping can be correlated to hand motion or that walking can be correlated to a specific combination of motions from the feet, arms and body. In this paper, we propose a new representation of human actions called Sequence of the Most Informative Joints (SMIJ), which is extremely easy to interpret. At each time instant, we automatically select a few skeletal joints that are deemed to be the most informative for performing the current action. The selection of joints is based on highly interpretable measures such as the mean or variance of joint angles, maximum angular velocity of joints, etc. We then represent an action as a sequence of these most informative joints. Our experiments on multiple databases show that the proposed representation is very discriminative for the task of human action recognition and performs better than several state-of-the-art algorithms.

Original languageEnglish
Title of host publicationIEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
Pages8-13
Number of pages6
DOIs
Publication statusPublished - 20 Aug 2012
Externally publishedYes
Event2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2012 - Providence, RI, United States
Duration: 16 Jun 201221 Jun 2012

Other

Other2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2012
CountryUnited States
CityProvidence, RI
Period16/6/1221/6/12

Fingerprint

Optical flows
Angular velocity
Bayesian networks
Classifiers
Trajectories
Experiments

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Electrical and Electronic Engineering

Cite this

Ofli, F., Chaudhry, R., Kurillo, G., Vidal, R., & Bajcsy, R. (2012). Sequence of the Most Informative Joints (SMIJ): A new representation for human skeletal action recognition. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (pp. 8-13). [6239231] https://doi.org/10.1109/CVPRW.2012.6239231

Sequence of the Most Informative Joints (SMIJ) : A new representation for human skeletal action recognition. / Ofli, Ferda; Chaudhry, Rizwan; Kurillo, Gregorij; Vidal, René; Bajcsy, Ruzena.

IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 2012. p. 8-13 6239231.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Ofli, F, Chaudhry, R, Kurillo, G, Vidal, R & Bajcsy, R 2012, Sequence of the Most Informative Joints (SMIJ): A new representation for human skeletal action recognition. in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops., 6239231, pp. 8-13, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2012, Providence, RI, United States, 16/6/12. https://doi.org/10.1109/CVPRW.2012.6239231
Ofli F, Chaudhry R, Kurillo G, Vidal R, Bajcsy R. Sequence of the Most Informative Joints (SMIJ): A new representation for human skeletal action recognition. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 2012. p. 8-13. 6239231 https://doi.org/10.1109/CVPRW.2012.6239231
Ofli, Ferda ; Chaudhry, Rizwan ; Kurillo, Gregorij ; Vidal, René ; Bajcsy, Ruzena. / Sequence of the Most Informative Joints (SMIJ) : A new representation for human skeletal action recognition. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 2012. pp. 8-13
@inproceedings{ec2ce631b90348afb586124f91056a3a,
title = "Sequence of the Most Informative Joints (SMIJ): A new representation for human skeletal action recognition",
abstract = "Much of the existing work on action recognition combines simple features (e.g., joint angle trajectories, optical flow, spatio-temporal video features) with somewhat complex classifiers or dynamical models (e.g., kernel SVMs, HMMs, LDSs, deep belief networks). Although successful, these approaches represent an action with a set of parameters that usually do not have any physical meaning. As a consequence, such approaches do not provide any qualitative insight that relates an action to the actual motion of the body or its parts. For example, it is not necessarily the case that clapping can be correlated to hand motion or that walking can be correlated to a specific combination of motions from the feet, arms and body. In this paper, we propose a new representation of human actions called Sequence of the Most Informative Joints (SMIJ), which is extremely easy to interpret. At each time instant, we automatically select a few skeletal joints that are deemed to be the most informative for performing the current action. The selection of joints is based on highly interpretable measures such as the mean or variance of joint angles, maximum angular velocity of joints, etc. We then represent an action as a sequence of these most informative joints. Our experiments on multiple databases show that the proposed representation is very discriminative for the task of human action recognition and performs better than several state-of-the-art algorithms.",
author = "Ferda Ofli and Rizwan Chaudhry and Gregorij Kurillo and Ren{\'e} Vidal and Ruzena Bajcsy",
year = "2012",
month = "8",
day = "20",
doi = "10.1109/CVPRW.2012.6239231",
language = "English",
isbn = "9781467316118",
pages = "8--13",
booktitle = "IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops",

}

TY - GEN

T1 - Sequence of the Most Informative Joints (SMIJ)

T2 - A new representation for human skeletal action recognition

AU - Ofli, Ferda

AU - Chaudhry, Rizwan

AU - Kurillo, Gregorij

AU - Vidal, René

AU - Bajcsy, Ruzena

PY - 2012/8/20

Y1 - 2012/8/20

N2 - Much of the existing work on action recognition combines simple features (e.g., joint angle trajectories, optical flow, spatio-temporal video features) with somewhat complex classifiers or dynamical models (e.g., kernel SVMs, HMMs, LDSs, deep belief networks). Although successful, these approaches represent an action with a set of parameters that usually do not have any physical meaning. As a consequence, such approaches do not provide any qualitative insight that relates an action to the actual motion of the body or its parts. For example, it is not necessarily the case that clapping can be correlated to hand motion or that walking can be correlated to a specific combination of motions from the feet, arms and body. In this paper, we propose a new representation of human actions called Sequence of the Most Informative Joints (SMIJ), which is extremely easy to interpret. At each time instant, we automatically select a few skeletal joints that are deemed to be the most informative for performing the current action. The selection of joints is based on highly interpretable measures such as the mean or variance of joint angles, maximum angular velocity of joints, etc. We then represent an action as a sequence of these most informative joints. Our experiments on multiple databases show that the proposed representation is very discriminative for the task of human action recognition and performs better than several state-of-the-art algorithms.

AB - Much of the existing work on action recognition combines simple features (e.g., joint angle trajectories, optical flow, spatio-temporal video features) with somewhat complex classifiers or dynamical models (e.g., kernel SVMs, HMMs, LDSs, deep belief networks). Although successful, these approaches represent an action with a set of parameters that usually do not have any physical meaning. As a consequence, such approaches do not provide any qualitative insight that relates an action to the actual motion of the body or its parts. For example, it is not necessarily the case that clapping can be correlated to hand motion or that walking can be correlated to a specific combination of motions from the feet, arms and body. In this paper, we propose a new representation of human actions called Sequence of the Most Informative Joints (SMIJ), which is extremely easy to interpret. At each time instant, we automatically select a few skeletal joints that are deemed to be the most informative for performing the current action. The selection of joints is based on highly interpretable measures such as the mean or variance of joint angles, maximum angular velocity of joints, etc. We then represent an action as a sequence of these most informative joints. Our experiments on multiple databases show that the proposed representation is very discriminative for the task of human action recognition and performs better than several state-of-the-art algorithms.

UR - http://www.scopus.com/inward/record.url?scp=84864972509&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84864972509&partnerID=8YFLogxK

U2 - 10.1109/CVPRW.2012.6239231

DO - 10.1109/CVPRW.2012.6239231

M3 - Conference contribution

AN - SCOPUS:84864972509

SN - 9781467316118

SP - 8

EP - 13

BT - IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops

ER -