Action Recognition OLD

Team: K. Guo, P. Ishwar, J. Konrad
Funding: National Science Foundation (CISE-SNC-NOSS)
Status: Ongoing (2007-…)

Background: The proliferation of network cameras in the last decade  has led to a surveillance video overload; cameras produce data at rates far exceeding the capacity of human operators. While recognizing events taking place in camera’s field of view is an essential step in security, human operator in the loop is unsustainable because of cost and fatigue factors. Clearly, an automatic recognition of events is essential from the standpoint of efficiency and cost. In addition to  applications in visual surveillance, action recognition can be a powerful tool for video indexing and retrieval as well.  However, action analysis is a challenging problem due to complexities present in the scene (e.g., clutter and occlusions), during acquisition (e.g., camera motion, dependence of object appearance on viewpoint) and within action itself (e.g., object non-rigidity, motion variability). Central to these challenges is action representation, i.e., the way one models actions in a video sequence.

Example of a human action sequence: three frames from a "jump-jack" action (top row) and corresponding silhouettes (bottom row)
Fig.1 Example of a human action sequence: three frames from a "jumping jack" action and corresponding silhouettes

Summary: In the current project, we perform action recognition from a video sequence captured by a typical surveillance camera. First, we apply background subtraction to all frames of the sequence in order to extract moving object’s silhouette (object tunnel). We then view an action as a temporal sequence of shape-deformations observed in object’s silhouette. In order to eliminate global motion of the silhouette, we align the centroids of silhouettes in all frames of the sequence.  We represent each action by the empirical covariance matrix of a13-dimensional vector of normalized geometric features that capture the shape of the silhouette tunnel. The similarity of two actions is measured in terms of a Riemannian metric between their covariance matrices.  We use the nearest neighbor rule for action classification.

In another part of this project, we are concerned with the detection of action boundaries. A surveillance camera produces continuous video that includes different types of actions, for example following one another. Since in our action recognition method the covariance model is applied to a certain number of frames, it can overlap  two different actions.  In this case,  our method produces inaccurate results. Thus, we need to identify temporal boundaries of actions prior to recognizing them; then our covariance model can be applied without straddling temporal action boundaries. To this effect, we use a non-parametric statistical framework to learn the distribution of the distance between covariance descriptors and detect action changes as covariance-distance outliers.

Each point s0 of a silhouette sequence has a 13-dimensional feature vector associated with it
Fig.2 Depiction of distances used to construct our 13-dimensional feature vector at each silhouette point.

Results:

Fig.1 above shows frames from a test video sequence with  corresponding silhouettes (from Weizmann Human Action Database) . Fig. 2 on right shows distances from an inner point of a silhouette to its boundary that are used to build a 13-dimensional vector of  spatio-temporal features for action recognition. Fig. 3 below shows each of the 13 features for a single silhouette frame as an intensity image. Note the different characteristics captured by each feature.

Individual components of a feature vector depicted as intensity images. The origin is at the top left corner and brighter points denote larger values.
Fig. 3 Individual components of a feature vector depicted as intensity images. The origin is at the top left corner and brighter points denote larger values.

The first experiment is based on Weizmann Human Action Database that contains 90 low-resolution video sequences (180×144 pixels, 50fps) that show 9 different people with each person performing 10 different actions, such as jumping, walking, running, skipping, etc. We measured the performance of our algorithm by using the leave-one-out cross validation (LOOCV) and the nearest-neighbor (NN) rule. We attained the correct classification ratio (CCR) of 97.05% for action segments of length N = 8 with 5-fold lower computational complexity than the method proposed by Gorelick et al. [PAMI-2007]. Furthermore, we achieves a 100% CCR by fusing segment-level decisions together using the majority rule in order to obtain a decision for the entire video clip.

The second experiment focuses on how to detect action changes, i.e., the temporal boundaries of actions without recognizing them. It is similar to scene-cut detection but in the space of actions.  Figs. 4 and Fig. 5 below show examples of action change detection for one individual (video sequence constructed by concatenating different actions by the same individual). There are two types of action detection errors: (i) false positive errors which occur when segments which have no action changes are classified as having action changes and (ii) false negative errors which occur when segments which have action changes are classified as having no action changes. The combined total number of action segments across all 9 test video sequences is 597 out of which 61 segments have action changes and 536 do not. The proposed action change detection method produces 1 false negative error (Percent false negative error PFN = 100/61 = 1.64%) and 1 false positive error (Percent false positive error PFP = 100/536 = 0.19%) for a
confidence level of 0.9. The values of PFN and PFP change by less than 1% for a range of confidence levels from 0.6 to 0.95.

Nearest-neighbor distances (top) and corresponding cumulative probability values (bottom) for consecutive action segments from ground-truth, multi-action video sequence built from from Weizmann Human Action Database for “Daria” (Fig. 1(a)). Groundtruth actions are shown schematically above the top plot. The circles mark correctly-identified action boundaries (center of a segment in which action change takes place), while the squares mark erroneous boundaries.
Fig. 4 Nearest-neighbor distances (top) and corresponding cumulative probability values (bottom) for consecutive action segments from ground-truth, multi-action video sequence built from from Weizmann Human Action Database. Groundtruth actions are shown schematically above the top plot. The circles mark correctly-identified action boundaries, while the squares mark erroneous boundaries.
Nearest-neighbor distances (top) and corresponding cumulative probability values (bottom) for consecutive action segments from time-continuous, camera-captured video (Fig. 1(b)). Groundtruth actions are shown schematically above the top plot. The circles mark correctly-identified action boundaries (center of a segment in which action change takes place).
Fig. 5 Nearest-neighbor distances (top) and corresponding cumulative probability values (bottom) for consecutive action segments from time-continuous, camera-captured video. Groundtruth actions are shown schematically above the top plot. The circles mark correctly-identified action boundaries.

Publications:

K. Guo, P. Ishwar, and J. Konrad, “Action change detection in video by covariance matching of silhouette tunnels,” in Proc. IEEE Int. Conf. Acoustics Speech Signal Processing, Mar. 2010 [PDF: 217KB].

K. Guo, P. Ishwar, and J. Konrad, “Action recognition from video by covariance matching of silhouette tunnels,” in Proc. Brazilian Symp. on Computer Graphics and Image Proc., Oct. 2009 [PDF: 325KB].