Behavior Subtraction

Team: P.-M. Jodoin, V. Saligrama, J. Konrad
Funding: National Science Foundation (CISE-CCF-CIF)
Status: Completed (2008-2012)

Background: In order to realize the goal of autonomous visual sensor network, i.e., camera network that does not require round-the-clock human operator involvement, video analysis methods need to be developed capable of detecting various activities, especially abnormalities. This is often referred to video as analytics. The simplest form of video analytics, background subtraction, may be sufficient for restricted-area access control where a mere detection of movement is sufficient. However, it may not be sufficient for the detection of unusual behavior. Thus, the question one needs to answer is: How to distinguish normal motion patterns from abnormal ones?

Summary: Using the background detection results, we have developed an abnormal behavior detection framework that can detect abnormally high activity, departure from typical activity, etc. all within the same formulation. By aggregating motion labels for a given pixel from all frames of a training sequence, we build a low-dimensionality representation against which we test an aggregate computed from the observed sequence. If the observed aggregate eitehr exceeds or departs from the training aggregate, abnormality is declared. This simple strategy leads to remarkably robust results as demonstrated in video sequences below.

Results: Below are shown some experimental results produced by our method. Each video below shows the original sequence (on left), the result of background substraction (right of the original video),  and the result of behavior subtraction tuned to detect abnormally high activity (righmost in 1×3 panel, bottom in 2×2 panel).

Commuter train next to MassPike: Although train motion is similar to the motion of cars, it is discovered as anomaly.

Suspicious pedestrian on MassPike overpass: A standing pedestrian is detected on the overpass despite small size.

Tram on Commonwealth Avenue: The tram travels alongside cars but is discovered as anomaly against regular traffic.

Fountain obstructing moving cars: Despite very unstructured movement of fountain water, it is classified as normal while a van behind is discovered as anomalous (note the occluding fountain that splits the van body in two).

Unstructured movement of water surface: Waves are classified as normal whereas circular rings due to stones thrown into the lake are classified as abnormal events.

Speed boat on Charles River: Unstructured waves on water surface are classified as normal whereas a fast-moving boat is classified as an abnormal event.

Abandoned object: The motor traffic is classified as as a normal event but an abandoned object is classified as abnormality.

Group of pedestrians: While each pedestrian is classified as a normal event, a group of pedestrians is considered to be an abnormality.

Publications:

• P.-M. Jodoin, V. Saligrama, and J. Konrad, “Behavior subtraction,” in Proc. SPIE Visual Communications and Image Process., vol. 6822, pp. 10.1-10.12, Jan. 2008.
• P.-M. Jodoin, J. Konrad, and V. Saligrama, “Modeling background activity for behavior subtraction,” in ACM/IEEE Int. Conf. Distributed Smart Cameras, Sept. 2008.
• P.-M. Jodoin, J. Konrad, V. Saligrama, and V. Veilleux-Gaboury, “Motion detection with an unstable camera,” in Proc. IEEE Int. Conf. Image Processing, Oct. 2008.
• V. Saligrama, J. Konrad, and P.-M. Jodoin, “Video anomaly detection: A statistical approach,” in IEEE Signal Process. Mag., vol. 27, pp. 18-33, Sept. 2010.
• P.-M. Jodoin, V. Saligrama, and J. Konrad,  “Behavior subtraction,” IEEE Trans. Image Process., vol. 21, pp. 4244-4255, Sept. 2012.