RAPiD: Rotation-Aware People Detection in Overhead Fisheye Images

Team: Zhihao Duan, M. Ozan Tezcan, Hayato Nakamura, Prakash Ishwar, Janusz Konrad
Funding: Advanced Research Projects Agency – Energy (ARPA-E)
Status: Ongoing (2019-…)

Background: Occupancy sensing is an enabling technology for smart buildings of the future; knowing where and how many people are in a building is key for saving energy, space management and security (e.g., fire, active shooter). In this work, we develop a fast people-detection algorithm for overhead fisheye images.

Summary: Our previous people-detection algorithm for fisheye images by Li et al. produced accurate results at very high computational complexity; even on a relatively small HABBOF dataset it required tens of GPU-hours during inference. The main reason for this is the fact that the algorithm applies YOLO (version 3) to each fisheye image up to 24 times.  In this work, we propose a faster and accurate approach by designing an end-to-end neural network, which extends YOLO v3 to precisely handle human-body orientation in overhead fisheye images.  Our fully-convolutional neural network directly regresses the angle of each bounding box using a periodic loss function, which accounts for angle periodicities. We show that our simple, yet effective method outperforms state-of-the-art results on three datasets: Mirror Worlds, HABBOF, and CEPDOF.

Technical Approach: RAPiD is a single-stage fully-convolutional neural network that predicts arbitrarily-rotated bounding boxes of people in a fisheye image. It extends the model proposed in YOLO, one of the most successful object detection algorithms for standard images. The network architecture diagram is shown below. In addition to predicting the center and size of a bounding box, RAPiD also predicts its angle. This is accomplished by a periodic loss function based on an extension of a common regression loss. This allows us to predict the exact rotation of each bounding box in an image without any assumptions and additional computational complexity. Unlike many previous methods, RAPiD is an end-to-end algorithm, so we can train or fine-tune its weights on annotated fisheye images. We show that such fine-tuning of a model trained on standard images significantly increases the performance. An additional aspect of this work, motivated by its focus on people detection, is the replacement of the common regression-based loss function used in multi-class object detection algorithms with single-class object detection. The inference speed of RAPiD is nearly identical to that of YOLO since it is applied to each image only once without the need for pre-/post-processing.

Datasets: Although there exist several datasets for people detection from overhead, fisheye images, either they are not annotated with rotated bounding boxes, such as in the Mirror Worlds dataset, or the number of frames and people are limited, such as in the HABBOF dataset. Therefore, we collected and labeled a new dataset named Challenging Events for Person Detection from Overhead Fisheye images (CEPDOF). The new dataset contains many more frames and human objects, and also includes challenging scenarios such as crowded rooms, various body poses, and low-light conditions, which are absent in other datasets. Furthermore, CEPDOF is annotated spatio-temporally, i.e., bounding boxes of the same person carry the same ID in consecutive frames, and thus can be also used for additional vision tasks using overhead, fisheye images, such as video-object tracking and human re-identification.

Since the original Mirror Worlds (MW) dataset is annotated only with axis-aligned bounding boxes that often do not fit tightly around people, we have manually annotated a subset of the MW dataset with rotated bounding-box labels, that we refer to as MW-R. For more details and to download both datasets, please refer to the links below.

• Challenging Events for Person Detection from Overhead Fisheye images (CEPDOF)
• Rotated Bounding-Box Annotations for Mirror Worlds Dataset (MW-R)