Character Recognition in Air-Writing Based on Network of Radars for Human-Machine Interface

Abstract

Radar technology plays a vital role in contact-less detection of hand gestures or motions, which forms an alternate and intuitive form of human–computer interface. Air-writing refers to the writing of linguistic characters or words in free space by hand gesture movements. In this paper, we propose an air-writing system based on a network of millimeter wave radars. We propose a two-stage approach for extraction and recognition of handwriting gestures. The extraction processing stage uses a fine range estimate combined with the trilateration technique to detect and localize the hand marker, followed by a smoothening filter to create a trajectory of the character through the hand movement. For the recognition stage, we explore two approaches: one extracts the time-series trajectory data and recognizes the drawn character using long short term memory (LSTM), bi-directional LSTM (BLSTM), and convolutional LSTM (ConvLSTM) with connectionist temporal classification (CTC) loss function, and the other approach reconstructs a 2D image from the trajectory of drawn character and uses deep convolutional neural network (DCNN) to classify the alphabets drawn by the user. ConvLSTM-CTC performs best among LSTM variants on time-series trajectory data achieving 98.33% classification accuracy similar to DCNN over the chosen character set. This paper employs real data using a network of three 60-GHz millimeter wave radar sensor to demonstrate the success of the proposed setup and associated algorithm with design consideration.