Occupancy sensing using thermal door sensors
Team: Mertcan Cokbas, Prakash Ishwar, Janusz Konrad
Funding: Advanced Research Projects Agency – Energy (ARPA-E)
Status: Completed (2018-2020)
Background: The knowledge of occupancy in a room can be useful for saving energy, space management and in emergency situations. Today, modern buildings are equipped with PIR sensors to detect occupancy but they are primarily used for lighting control. While such sensors can be used for heating, ventilation and air conditioning (HVAC) control, they offer only limited energy savings (if a single person is present in a classroom, HVAC gets engaged as if fully-occupied). In order to offer better energy savings, a system is needed that can estimate occupancy accurately (e.g., 50% of room’s capacity). In this work, we focus on counting the number of people in a room by using thermal sensors mounted above doors..
Summary: Over the years, numerous people-counting systems have been proposed leveraging various sensing modalities, e.g., surveillance cameras, MAC address trackers, WiFi signal measurement, CO2 sensors and thermal sensors each with its own deficiencies. For instance, surveillance cameras may not be acceptable in scenarios where privacy is expected, MAC address trackers require people to carry a networked device, WiFi signal measurement is sensitive to EM interference and unreliable for crowds of people, while CO2 sensors have delayed reaction times due to slow mixing of gases. On the other hand, thermal sensors do not suffer from these issues. Although people-counting systems based on thermal sensors have been proposed in the past, they all point sensors into a room thus requiring many sensors to cover large rooms. Instead, we propose to estimate room’s occupancy by mounting sensors above entry/exit points of a room and detecting the change in occupancy (people entering and leaving the room). Clearly, this limits the number of sensors needed regardless of room size. The room’s occupancy is inferred the detected number of entries and exits assuming that the initial occupancy is known.
Technical Approach: In our approach, we analyze consecutive thermal frames captured by a sensor mounted above a door in three steps: (1) background subtraction to detect the presence of one or more people under the sensor; (2) event detection to identify the beginning and end of an entry or exit event spanning multiple thermal frames; and (3) event classification as either an entry or exit. We propose two different algorithms for event detection and classification. The baseline algorithm assumes only one person passes through the door at a time. The multi-person algorithm can handle multiple individuals passing through the door simultaneously.
TIDOS Dataset: In order to thoroughly evaluate the performance of our algorithms, we collected a dataset of thermal sequences, to the best of our knowledge the first of its kind. Our dataset, Thermal Images for Door-based Occupancy Sensing (TIDOS), features 6 thermal recordings with about 118,000, 32×24-pixel frames in total, and the associated manually determined labels: initial people count for each recording and people count in the room for each frame. The people count changes occur in frames when a person leaves a door frame and enters a room (count increment by 1) or leaves a room (count decrement by 1). Details of the dataset and instructions how to download can be accessed via this link:
Experimental Results: We compared the performance of both algorithms on all 6 sequences of our dataset. The quantitative results can be seen in the table below for three evaluation metrics:
- Mean Absolute Error (MAE) between the true an estimated people count,
- Per-Person Mean Absolute Error (MAEpp) – MAE normalized by the average people count of a recording,
- Windowed Count-Change Correct Classification Rate (CCRWCC) – unlike the two metrics above this metric is not sensitive to cumulative errors (for definition, please see the paper below and this page for a detailed description and analysis).
As it can be seen from the table, the multi-person algorithm outperforms the baseline algorithm on all thermal recordings. It is clear that the multi-person algorithm outperforms the baseline algorithm on all recordings, often significantly, except for Lunch Meeting 2 in which both algorithms have the same CCRWCC but the baseline algorithm has a slightly lower MAE. This is due to the fact that Lunch Meeting 2 contains only single-person events, i.e., one person passes through the door at a time. The reason for the slightly lower MAE of the baseline algorithm in Lunch Meeting 2 is that its event detection has slightly more accurate event timing than for the multi-person algorithm. Please note that the 6 recordings have different challenges that are described in detail in the TIDOS dataset (link above).
Publications:
- M. Cokbas, P. Ishwar, and J. Konrad, “Low-resolution overhead thermal tripwire for occupancy estimation,’‘ in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Perception Beyond Visible Spectrum (PBVS) Workshop, June 2020.