Virtual Keyboard

Team: F. Yi, J. Konrad
Funding: Self funded
Status: Completed (2011-2012)

Background: Although keyboard and mouse have been essential in human-computer communication for decades, certain new application scenarios require a new interface modality.Consider a “wet lab” scenario where a researcher, equipped with latex gloves, works on bench with biological materials, chemicals agents, and other contaminants, but at the same time must interact with a computer, e.g., to mark the completion of various steps, input amounts of chemicals used, etc. Clearly, each time the researcher uses keyboard/mouse he/she must remove the gloves to avoid keyboard contamination, which slows down the experiment and creates waste (latex gloves disposed off). Thus, of interest is a computer interface in form of a “virtual keyboard/mouse” where the researcher can enter alphanumeric symbols and draw shapes without the need to use a physical device.

Summary: We developed a virtual keyboard and mouse using Microsoft Kinect camera placed above a “wet” bench. The camera is mounted above the bench and “looks” down at the bench surface (plane) that can be touched by a finger in order to enter an alphanumeric character or can be swiped by a finger in order to draw a shape. In our virtual keyboard, we exploit Kinect’s ability to accurately compute depth of various objects in its field of view. First, we calibrate our algorithm by capturing the depth of the bench void of any hands or other objects, the so-called background depth. Then, we perform “depth subtraction”, i.e., we subtract the current depth captured by the Kinect from the background depth captured at calibration, to obtain a relative depth of objects with respect to the depth of the bench. Pixels with relative depth smaller than a threshold are close to the bench, but form irregular patches. In the last step, we clean up those patches by simple morphological processing, remove very small patches and find the center of each patch which identifies the contact point of a finger with the bench surface. A computer monitor mounted above the bench shows either a regular QWERTY keyboard, a numeric keypad or a drawing area (touch pad), depending on user selection, along with user hand moving across the screen. When a finger is located above a key of interest, the user can gently tap the bench to input the specific number or character. The user can also draw lines, curves and other shapes by selecting the touch pad option.

Results: Our virtual keyboard has been tested by students in the department and received very positive comments. However, no formal quantitative testing has been performed. The video below shows the virtual keyboard in action. Note that both keyboards as well as the touch pad can be resized  by familiar zoom-in/zoom-out pinch performed by both hands (spread the hands out/bring them together).