Sequence of the most informative joints (SMIJ): A new representation for human skeletal action recognition

Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, René Vidal, Ruzena Bajcsy

Research output: Contribution to journalArticle

181 Citations (Scopus)


Much of the existing work on action recognition combines simple features with complex classifiers or models to represent an action. Parameters of such models usually do not have any physical meaning nor do they provide any qualitative insight relating the action to the actual motion of the body or its parts. In this paper, we propose a new representation of human actions called sequence of the most informative joints (SMIJ), which is extremely easy to interpret. At each time instant, we automatically select a few skeletal joints that are deemed to be the most informative for performing the current action based on highly interpretable measures such as the mean or variance of joint angle trajectories. We then represent the action as a sequence of these most informative joints. Experiments on multiple databases show that the SMIJ representation is discriminative for human action recognition and performs better than several state-of-the-art algorithms.

Original languageEnglish
Pages (from-to)24-38
Number of pages15
JournalJournal of Visual Communication and Image Representation
Issue number1
Publication statusPublished - 1 Jan 2014
Externally publishedYes



  • Bag-of-words
  • Berkeley MHAD
  • Cross-database generalization
  • HDM05
  • Human action recognition
  • Human action representation
  • Informative joints
  • Linear dynamical systems
  • MSR Action3D
  • Normalized edit distance

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Media Technology
  • Computer Vision and Pattern Recognition
  • Signal Processing

Cite this