Thermal Images for Door-based Occupancy Sensing (TIDOS)

Motivation

Although counting people indoors can be peformed using surveillance cameras, MAC address trackers, WiFi signal measurement or CO₂ sensors, each has its own deficiencies. On the other hand, thermal sensors do not violate privacy, do not require carrying a tracker, are robust to EMF interference and instantaneously react to body temperature. Although several systems using thermal sensors for people counting have been proposed in the literature, no thermal datasets have been published to the best of our knowledge to date. Therefore, we introduce Thermal Images for Door-based Occupancy Sensing (TIDOS) dataset containing 6 thermal recordings in various occupancy scenarios and associated manual annotations of the number of people in a room.

Description

All recordings have been performed in a small classroom with two doors on Boston University campus. One Melexis MLX90640 temperature sensor, connected to Raspberry Pi 3 B+ for data collection, was mounted about 8 inches above each door and pointed down. A typical thermal frame (32×24 pixels) captured by one of the sensors of a person leaving the classroom is shown above. All recordings were made at 16 frames per second in various entry/exit scenarios and a range of challenges, such as lingering in the door frame, quickly running through the door, wearing different types of clothing, multiple people going through the door at the same time, passing through the door while carrying various items, etc. A more detailed description of each recording is provided in the table below.

Thermal Recording	No. of frames	No. of entries and exits	Initial people count	Challenges (scenario)
Lecture	7,520	2	9	Lingering in doorway (only single-person events)
Lunch Meeting 1	37,536	25	0	Wearing a coat; carrying various items; multiple people passing through at the same time
Lunch Meeting 2	9,344	8	12	Carrying a backpack (only single-person events)
Lunch Meeting 3	28,128	69	7	Lingering in doorway; wearing a hoodie or carrying a backpack; two people standing in a door and handshaking; multiple people passing through at the same time
Edge Cases	13,120	24	6	Long lingering in doorway; one or two people standing in a door while another person is passing through; multiple people passing through at the same time
High Activity	22,560	133	4	Wearing a hoodie or thick coat; carrying a backpack; pushing a chair through doorway; leaning against a closed door; one person standing in a door while another one is passing through; multiple people passing through at the same time

Data Format

Since the data were recorded in a classroom with two doors (two sensors), each recording consists of two thermal sequences (one for each sensor), each stored in a separate folder. Each folder contains JSON files, each file storing 1 second of thermal data; the number of JSON files in a folder equals the length of the thermal sequence in seconds.

All JSON files have the following attributes: [Frames_A, Frames_B, Frames_C, Hit, TimeStamp], where all thermal data are stored in Frames_B. In this particular dataset, Frames_A and Frames_C are not used and, therefore, are always empty. Frames_B contains a 16×768 double matrix, each row of this matrix corresponding to one frame (32 x 24 = 768) of temperature values. The data were recorded at 16 Hz, therefore the Frames_B array has 16 rows. The attribute Hit also carries no information; it was used for verifying data consistency during recording. The attribute TimeStamp stores the time of recording the first of 16 frames (in epoch form). All data were recorded in GMT-4 time zone.

Data Labeling

Assuming an initially-empty room (initial people count = 0), each thermal frame was manually labeled with the current number of people in this room. The people-count changes were judged to occur in frames when a person leaves the door frame and enters the room (people-count increment by 1) or leaves the room (people-count decrement by 1). In other words, if a person is entering and still visible in the field of view of the sensor, the people count is not changed until the person leaves the frame completely. Clearly, in some scenarios this people count may be negative. In order to obtain the true number of people in each room, one needs to add the initial people count in this room (prior to the beginning of the recording) provided in the table above.

Labels for each thermal sequence are stored in a .mat file, with two such files for each recording (one for each door sensor). Each .mat file is a vector of length equal to the number of frames in a thermal sequence.

Dataset Download

You may use this dataset for non-commercial purposes. If you publish any work reporting results using this dataset, please cite the following paper:

M. Cokbas, P. Ishwar, and J. Konrad, “Low-resolution overhead thermal tripwire for occupancy estimation,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Perception Beyond Visible Spectrum (PBVS) Workshop, June 2020.

To access the download page, please complete the form below (tested only in Chrome).

TIDOS Download Form

Name*
First Last
Institution*

Email*

Contact

Please contact [mcokbas] at [bu] dot [edu] if you have any questions.

Acknowledgements

The development of this dataset was supported in part by the Advanced Research Projects Agency – Energy (ARPA-E), within the Department of Energy, under agreement DE-AR0000944.

We would also like to thank Mr.~Yu Xiao (MS ECE, 2019) for developing a dataset annotation tool and for annotating a large portion of the dataset, and also other students at Boston University for their participation in the recording and annotation of this dataset.