Behavior Subtraction

Team: P.-M. Jodoin, V. Saligrama, J. Konrad
Funding: National Science Foundation (CISE-CCF-CIF)
Status: Ongoing (2008-…)

Background: In order to realize the goal of autonomous visual sensor network, i.e., camera network that does not require round-the-clock human operator involvement, video analysis methods need to be developed capable of detecting various activities, especially abnormalities. This is often referred to video as analytics. The simplest form of video analytics, background subtraction, may be sufficient for restricted-area access control where a mere detection of movement is sufficient. However, it may not be sufficient for the detection of unusual behavior. Thus, the question one needs to answer is: How to distinguish normal motion patterns from abnormal ones?

Summary: Using the background detection results, we have developed an abnormal behavior detection framework that can detect abnormally high activity, departure from typical activity, etc. all within the same formulation. By aggregating motion labels for a given pixel from all frames of a training sequence, we build a low-dimensionality representation against which we test an aggregate computed from the observed sequence. If the observed aggregate eitehr exceeds or departs from the training aggregate, abnormality is declared. This simple strategy leads to remarkably robust results as demonstrated in video sequences below.

Results: Below are shown some experimental results produced by our method. Each video below shows the original sequence, the result of background substraction,  and the result of behavior subtraction tuned to detect abnormally high activity. Click on any of the images below to play a video file in Windows Media format (.wmv).  Mac users may need to download Windows Media® Components for QuickTime.


Commuter train next to MassPike (4MB): Although train motion is similar to motion of cars on the highway, it is discovered as anomaly.


Suspicious pedestrian on MassPike overpass (12MB): A suspicious pedestrian is discovered on I-90 overpass despite small size.


Tramway on Commonwealth Avenue (16MB): The tram travels alongside cars but is discovered as anomaly against regular traffic.


Fountain obstructing moving cars (8MB): Despite very unstructured movement of water in the fountain, it is classified as normal while a van behind is discovered as anomalous (note the occluding fountain that splits the van body in two)


Unstructured motion of water surface (6MB): Waves are classified as normal whereas circular rings due to stones thrown into the lake are classified as abnormal events.




Speed boat on Charles River (7MB): Unstructured waves on water surface are classified as normal whereas a fast-moving boat is classified as an abnormal event.




Abandoned object (9MB): The motor traffic is classified as as a normal event but an abandoned object is classified as abnormality.




Group of pedestrians (26MB): While each pedestrian is classified as a normal event, a group of pedestrians is considered to be an abnormality.