Head-mouse: A simple cursor controller based on optical measurement of head tilt
Ali HeydariGorji, Seyede Mahya Safavi, Cheng-Ting Lee, Pai H. Chou
HHEAD-MOUSE: A SIMPLE CURSOR CONTROLLERBASED ON OPTICAL MEASUREMENT OF HEAD TILT
Ali HeydariGorji , Seyede Mahya Safavi , Cheng-Ting Lee , Pai H. Chou Department of Electrical Engineering and Computer Science, University of California, Irvine, USA Department of of Computer Science National Tsing Hua University, Hsinchu City, 30013 Taiwane-mail: { heydari, safavis } @uci.edu, { davis0704.lee } @gmail.com, { phchou } @cs.nthu.edu.tw Abstract —This paper describes a wearable wireless mouse-cursor controller that optically tracks the degree of tilt of theuser’s head to move the mouse relative distances and thereforethe degrees of tilt. The raw data can be processed locally onthe wearable device before wirelessly transmitting the mouse-movement reports over Bluetooth Low Energy (BLE) protocolto the host computer; but for exploration of algorithms, the rawdata can also be processed on the host. The use of standardHuman-Interface Device (HID) profile enables plug-and-playof the proposed mouse device on modern computers withoutrequiring separate driver installation. It can be used in twodifferent modes to move the cursor, the joystick mode and thedirect mapped mode. Experimental results show that this head-controlled mouse to be intuitive and effective in operating themouse cursor with fine-grained control of the cursor even byuntrained users.
I. I
NTRODUCTION
Pointing on a two-dimensional display has been a fun-damental operation in graphical user interface in the pastthree decades and will continue to remain important in theforeseeable future. Pointing devices on a two-dimensionalscreen may have involved from desktop mouse and trackballs to track pads and touch screens. However, for patientssuffering from amputation or paralysis, alternative mechanismsmust be used to enable their operation with computers.Assistive technologies developed for enabling hands-freecursor control are mainly divided into eye trackers [1], headtrackers, tongue trackers [2], brainwave (EEG) sensors [3], andmuscle tension sensors (EMG) [4]. These techniques targetusers with a wide range of motor disabilities, and they haveall been shown to be effective in their own specific ways.However, they all come at a relatively high cost. Camera-based systems require continuous image processing, whichrequires nontrivial computation power and can operate forat most hours on a battery. This makes them unsuitable forportable long-term usage in the range of weeks or months [5].Tongue trackers may require special sensors to be placed in themouth and may be considered intrusive even if not invasive. Inaddition to a poor performance in terms of speed and accuracy,EEG electrodes are extremely complex and uncomfortable dueto the large number of electrodes and wires [6]. Althoughvoice recognition can handle text input for verbal commandsand dictation with out any body mounted extra device, it hasa low degree of freedom. In addition, the computational load due to voice processing unit makes it power consuming andslow.We envision a low-cost system that is comfortable to wear,intuitive to use, and require little or no training or calibration.To achieve these goals, we propose a simple mechanismthat requires minimal processing based on infra-red LED (IRLED) and photo detector pair. IR LED and photo detectorpair can be an effective mechanism for precise short-rangedistance measurement. The novelty with our work is theadaptation of the active IR-based mechanism to measuringdistance from the collar area to the sides of the chin as ourway of tracking head tilt to control the mouse cursor. TheIR LED and photo diodes are among the lowest-cost sensors,and the rest of the functionality including the ADC, processor,and wireless communication can all be done in commodityBluetooth Low Energy (BLE) microcontrollers on the orderof $2-3 in low-volume quantities. Due to the simplicity ofalgorithms and calculations, they can be implemented on theMCU to eliminate the need for a third party software on thehost system. As a result, the device can be directly connectedto the host system using generic HID drivers. It can also drivesthe cursor in two different modes for the ease of use. Bothmodes, the joystick and direct mapping, are explained in thefurther sections. We have built a prototype of the proposed“head mouse” system and tested it on a number of users.Experimental results show that our system to be intuitive touse, comfortable to wear, low power, and low cost.II. S
YSTEM D ESIGN
Fig. 1 shows the block diagram of the proposed system.The wearable system is centered around a microcontroller unit(MCU), sensors, a power source, and a wireless communi-cation interface. The wearable system transmits either rawor processed data to the host computer where the cursor isdisplayed.
A. MCU and Wireless Communication
We suggest an MCU-based embedded system. On oneside, it reads the signal from the sensor for detecting thehead position. The MCU is programmable and can processdata. On the other side, the MCU is connected to a wirelesscommunicating module to transfer the raw or processed datato the host system. Many conventional MCUs can be used. To a r X i v : . [ c s . H C ] J un e suitable for a wearable device, we choose one with more in-tegrated features, including on-chip analog-to-digital converter(ADC) for interfacing with the optical sensors, and low powerconsumption, rather than one with high performance.The choice of wireless communication enables ease of useand mobility of device. The device and the host need to sup-port a compatible protocol. We chose Bluetooth Low EnergyTechnology (BLE), suitable for low-data-rate communication.It consumes considerably less power compared to convectionalBluetooth modules. Moreover, BLE is also integrated on anumber of popular MCUs, making it low cost and compact.Our choice of MCU is the TI CC2541, although many otherMCUs in the similar class can also be used. The processor isan 8051-compatible core surrounded by 8-KB SRAM, 256 KBflash, 8-channel ADC, timers, UARTs, I2C, and SPI on thechip. It also includes a BLE transceiver that enables connectionto any Bluetooth device including the users’s computer. Weused this SoC to transmit data to the host system. B. Optical Sensor
Our proposed system consists of two pairs of IR LED andphotodiodes to estimate the distance of the LED from theedges of user’s chin. The collected data is digitized usingan embedded analog-to-digital converter and sent through theBLE module to the host computer. This subsection describesthe sensors in detail.The IR transmitters and receivers work on 940 nm wave-length and are available commercially off-the-shelf. For sim-plicity, the transmitter is set to be voltage-driven rather thancurrent-driven using a 220 Ω shunt resistor to control thecurrent and IR brightness as shown in Fig. 1. The receiveris an IR photo-detector, modeled as a variable resistor. Theresistance is determined by the amount of IR that the photo-detector receives. A shunt resistor of 22 K Ω is used to biasthe output. The output voltage changes approximately linearlywith respect to the intensity of the IR light. The IR intensityitself is a function of the distance between the sensor boardand the reflective surface (chin). Each sensor can detect thechin distance in the range of 0.2 to 4 inches.In order to make the device power efficient, pulse-widthmodulation (PWM) is used. Sensors are powered on onceevery 50 ms for a period of 1 ms; in other words, 20 Hzsampling frequency and 5% duty cycle. This reduces powerusage considerably and increases battery endurance withoutsignificantly affecting the device’s performance. Since theoutput signal is analog, it can have any value in the range ofVCC=3.3 volts and GND. The signal is converted using the on-chip ADC on the MCU. The ADC samples each photo diodeonce every 50 ms at 8-bit resolution using a time-multiplexingscheme. C. Power Subsystem
The power subsystem consists of a battery and a powerregulator. We chose a rechargeable lithium ion battery as thesource of power. The battery power is fed to a buck converter,the TPS62740DSS, to reduce 3.8 V to 2.5 V to be used by
PWMADCPD1IR LED1 PD2IR LED2
CPU Bluetooth
VccVccR1 R2 R3 R4 Fig. 1. The system block diagram. the MCU. Note that IR transmitters are directly connected to3.8 V from the battery (and GPIO pins control the path toground), rather than the regulated 2.5 V, as the higher voltageresults in more accurate distance measurement. The battery canbe charged by wired connectors using off-the-shelf chargers.Wireless charging can be considered for future work.
D. Mechanical Design
All of the components including the sensors, main board,and the battery are placed on a curved-shaped deck that caneasily be mounted on the neck as in Fig. 2. The overall size ofthe device is 4 × × . To make the device compatible fordifferent neck sizes, two fabric straps are mounted on eitherside of the deck. E. Host Computer
Depending on the mode of operation, further processing onthe raw data may be performed before transmission to thehost. For now, raw data is being sent directly and all theprocessing and algorithms are implemented on the host systemfor easier exploration and better reproducibility, as we can runa variety of algorithms on the same recorded data. We havedeveloped a Python-based Graphical User Interface (GUI) forPC to receive data and draw the correspond position on thescreen. We have also implemented the human interface device(HID) profile over BLE. The HID profile enables our device towork as a wireless mouse over BLE and enables plug-and-playoperation without requiring driver installation. A later versionwill process the data on the device and send mouse movementdata to the host just as a mouse would.III. M
ETHODS
The proposed algorithm for controlling a cursor on a screenis based on the distance between the IR sensor and the user’shead (chin). In resting position, two sensors approximatelyreceive the same amount of IR signal due to symmetry.Tilting down the user’s head brings both sensors closer to the ig. 2. The location of the optical sensors and the CC2541 module includingthe BLE, ADC, and the battery. A US quarter coin is placed for sizecomparison but is not part of the system
Pitching forward
Pitching backward
Yawing left
Yawing right
Resting position
Resting position
Resting position
Resting position
Resting position
Pitching backward
Fig. 3. The signal changes during the pitching and yawing movement. surface of reflection, resulting in a stronger signal. Likewise,tilting up the head weakens the reflection received by bothsensors due to the longer distance. When the head turns tothe right, the left sensor sees a weak reflection from the leftside of the chin, but the right sensor sees little change in thereflection since the chin is still above the receiver. Thus, aweak signal on the left sensor and a strong signal on the rightsensor indicates a cursor movement to right. The same isapplied to yawing to the left. Fig. 3 shows the signal valuesfor different movements of pitching up and down and turningto the left and right.The input data is filtered using moving average windowmethod in which the input data is the average of last 15 datasamples from the sensors. Averaging eliminates the ambientnoise and helps the cursor to move smoothly on the screen,preventing it from unwanted jumps. Also, due to the nature ofIR radiation, input noise level can vastly change. For example,the IR level in the day or under the sunlight is significantlyhigher than indoor spaces. To address this issue, we useddynamic thresholds to eliminate the problem of noise levelchanges. The signals go through a series of simple processingto map the head movement to cursor movement. Two differentmodes of joystick and direct mapping are implemented.
A. Joystick Mode
In joystick mode, the cursor moves only horizontally andvertically with a predefined constant speed. The data is usedto recognizes the direction of the movement. For each sensor,a lower and an upper threshold are adopted during calibrationtime at the initial 0.1 second of performance where user main-tains the resting head position. These thresholds are denoted by
Algorithm 1
Joystick calibration: Adjusting the thresholds. • Collect the samples during the calibration phase. Denotethe average of the samples by S1 initial and S2 init • Set th lower ← S1 init − th lower ← S2 init − th upper ← S1 init + th upper ← S2 init + Algorithm 2
Joystick mode. • If (cid:0) S1 ( t ) > th ( ) upper and S2 ( t ) > th ( ) upper (cid:1) Y ← Y − Y is the vertical cursor coordinate. Move thecursor one pixel downward. • If (cid:0) S1 ( t ) < th ( ) lower and S2 ( t ) < th ( ) lower (cid:1) Y ← Y +
1; Move the cursor one pixel downward. • If (cid:0) S1 ( t ) > th ( ) upper and S2 ( t ) < th ( ) lower (cid:1) X ← X −
1; X is the horizontal cursor coordinate. Move thecursor one pixel to the left. • If (cid:0) S1 ( t ) < th ( ) lower and S2 ( t ) > th ( ) upper (cid:1) X ← X +
1; Move the cursor one pixel to the right. th ( ) lower , th ( ) upper , th ( ) lower , th ( ) upper , where the superscripts 1 and 2represent the values for sensors 1 and 2. Algorithm 1 shows thepseudo code for adjusting the upper and lower thresholds forboth the sensors. The average of the raw data from the sensorsin the calibration mode is denoted by S1 initial and S2 initial . Thiscalibration helps us adjust the thresholds based on the ambientlight and IR noise in different situations. Once the thresholdsare learned, the cursor moves based on the algorithm describedin algorithm 2.We tried to follow a pre-drawn pattern on the screen and reacha target as depicted in Fig. 4. The path traveled by the cursorwhich is controlled by the user is drawn in a black line. InFig. 4 (a), the user is following the path in red, while in Fig. 4(b), the user tries to move the cursor inside the target depictedwith a square. To qualify the test scenario as successful, thecursor should stay inside the target region for two seconds. B. Direct-Mapping Mode
In direct-mapping mode, the absolute value of the sensordata is directly mapped to the cursor location on the GUI. Ifthe user tilts the head up to the maximum extend, the cursortouches the upper border. Similarly, the extreme positions ofthe user’s head in tilting down and yawing to the right and leftare mapped to the borders of the GUI screen. Thus, we needto define the range of head movement so that we can map itto the pointer location. A set of training movements should beconducted in the calibration phase at the first 4 seconds of theperformance. The user is required to maximally move his headto up, down, left, and right at startup. During this time, sensorsdata is being constantly checked to update the thresholds. Inthis mode, four thresholds are adopted for each sensor. Th ( ) U , Th ( ) D , Th ( ) L , and Th ( ) R are specifically adopted for sensor1in moving up, down, left, and right directions respectively.Likewise Th ( ) U , Th ( ) D , Th ( ) L , and Th ( ) R are learned for sensor 2. a) (b)(c) (d)Fig. 4. The testing platforms, (a) User tries to follow a predefined path in thejoystick mode. (b) User tries to guide the cursor into a target in the joystickmode. (c) Following the path in direct mapping mode. (d) Guiding the cursorto the target in direct mapping mode. Algorithm 3
Direct mapping mode, calibration phase. • While in Calibration do: • If (cid:0) S1 ( t ) < Th ( ) U and S2 ( t ) < Th ( ) U (cid:1) Update Th U ← S1 ( t ) Th U ← S2 ( t ) • If (cid:0) S1 ( t ) > Th ( ) D and S2 ( t ) > Th ( ) D (cid:1) Update Th D ← S1 ( t ) Th D ← S2 ( t ) • If (cid:0) S1 ( t ) > Th ( ) L and S2 ( t ) < Th ( ) L (cid:1) Update Th L ← S1 ( t ) Th L ← S2 ( t ) • If (cid:0) S1 ( t ) < Th ( ) R and S2 ( t ) > Th ( ) R (cid:1) Update Th R ← S1 ( t ) Th R ← S2 ( t ) Algorithm 3 shows the pseudo code for learning the thresholdsin direct mapping calibration phase.After 4 seconds, thresholds are fixed and put in to theequations. There are three equations, one governs the changesin Y coordinate of the cursor and the other two are used todetermine X coordinate. For Y we have:Y = (
Yaxis × avg ( S ( t ) , S ( t )) (cid:0) avg ( T h ( ) U , T h ( ) U ) − avg ( T h ( ) D , T h ( ) D ) (cid:1) (1)where Yaxis is the number of pixels in Y direction and avg () finds the average its inputs. For X coordinate, we had to usetwo equations. One controls the position in the left half of thescreen and the other one controls it in the right half of the screen. X = − ( S1 ( t ) − S2 ( t )) (cid:16) Th ( ) L − Th ( ) L (cid:17) × Xaxis = ( S − S )( T h ( ) R − T h ( ) R ) × ( Xaxis ) + Xaxis
C. Performance Metrics
According to Fitt’s law, there is an inherent trade-offbetween the speed and accuracy. We will resort to the per-formance metrics mostly used in the state-of-art literature toquantify the performance. These metrics including, the indexof difficulty, path efficiency, throughput, and overshoot aredefined as follows:
Index of Difficulty (ID) : ID is defined as ID = DW , (4)where D is the distance between the original location of thecursor and the center of the target, and W is the width of thetarget. Path Efficiency (PE) : Path efficiency is a measure of straight-ness of the path the cursor is traveling and is defined as theratio of the total distance between the original location of thecursor and target to the path length traveled by the cursor to : PE = DP , (5)where P is the length of the path traveled by the cursor. Throughput (TP) : Throughput is a metric for measuring howfast the cursor is moving and is defined as: TP = IDMT , (6)where MT is the time it takes for the user to move the cursorto the target zone. IV. R ESULTS
Two different users volunteered to test the implementedsystem in the two modes. The task requires the users to movethe cursor into a target. The location of the target on the screenand its width is randomly chosen, and the task is repeated. Infact, the index of difficult is a function of the location and thesize of the target. The GUI automatically keeps the records ofthe moving time ( MT ) and the path length P . The number oftrials for both joystick mode and direct mapping mode is 50each.Fig. 6 shows the path efficiency of the users for both the joy-stick and direct mapping modes, while Fig. 6 demonstrates thethroughput. It is observed in the path efficiency and throughputare significantly better in the joystick mode compared to thedirect mapping mode. This is observable from Fig. 4 wherethe user has a better controllability in joystick mode comparedto direct mapping. .8 1.9 2 2.1 2.2 2.3 2.4 2.5 2.6 2.7 Index of Difficulty P a t h e ff i c i en cy i n pe r c en t User1, joystickUser2 ,joystickUser1, direct mappingUser2, direct mapping
Fig. 5. Path efficiency with respect to index of difficulty for two modes ofjoystick and direct mapping.
Index of Difficulty T h r oughpu t ( p i x e l / s e c ) User1, joystickUser2 ,joystickUser1, direct mappingUser2, direct mapping
Fig. 6. Throughput with respect to index of difficulty for two modes ofjoystick and direct mapping.
V. C
ONCLUSIONS
We have presented a new wireless wearable device forcontrolling a pointing device by head movements. It usesconventional IR sensors to sense head movements, and acommodity MCU to map the sensor data to either relativemovement (joystick mode) or absolute location (direct map-ping mode) on the screen. While the principle of operation isrelatively straightforward, in practice, the system must addressseveral challenges, including optical noises from differentsources, the wide range of head movement on different users,accuracy, and power consumption. We solve these problems by using adaptive calibration for sensors, algorithm for co-ordinate generation, and PWM control. Several directions forfuture development remain. We plan to continue improvingaccuracy by better filtering of input signals and more tuningof threshold. For now, the head needs to have a considerablerotation to detect movement that causes a feeling of tiredness.By increasing the accuracy, we can reduce the range of headmovement, resulting in a better and more comfortable userexperience. Adding a clicking procedure based on a specificpattern of movement is also considered. For technical aspects,we aim to improve power efficiency and enable plug-and-playby performing calculations on the device and communicatingwith the host computer directly using HID profile. It willenable the user not only to control a general-purpose PC butalso embedded devices that can assist the user with a varietyof tasks, including navigating a wheel chair, a robot arm, andmany other assistive technologies to be invented.A
CKNOWLEDGMENT
The authors would like to thank Cho-An Lin and Hsun-FaZhou at National Tsing Hua University for their help with theBLE communication between the device and the host.R
EFERENCES[1] D. W. Hansen and Q. Ji, “In the eye of the beholder: A survey of modelsfor eyes and gaze,”
IEEE Trans. Pattern Anal. Mach. Intell , vol. 32, p.478500, Mar. 2010.[2] X. Huo, J. Wang, and M. Ghovanloo, “A magneto-inductive sensorbased wireless tongue-computer interface,”
IEEE Transactions on NeuralSystems and Rehabilitation Engineering , vol. 16, no. 5, pp. 497–504, Oct.2008.[3] D. J. McFarland, D. J. Krusienski, W. A.Sarnackia, W.A., and J. R.Wolpaw, “Emulation of computer mouse control with a noninvasivebraincomputer interface,”
Journal of neural engineering , vol. 5, no. 2,p. 101, 2008.[4] M. R. Williams and R. F. Kirsch, “Evaluation of head orientation andneck muscle EMG signals as command inputs to a humancomputerinterface for individuals with high tetraplegia,”
IEEE Transactions onNeural Systems and Rehabilitation Engineering , vol. 16, no. 5, pp. 485–496, Oct. 2008.[5] A. Al-Rahayfeh and M. Faezipour, “Eye tracking and head movement de-tection: A state-of-art survey,”
IEEE Journal of Translational Engineeringin Health and Medicine , vol. 1, pp. 2 100 212–2 100 212, 2013.[6] C. McCrimmon, J. Fu, M. Wang, L. Silva Lopes, P. Wang;, A. Karimi-Bidhendi, C. Liu, P. Heydari, Z. Nenadic, and A. Do, “Performanceassessment of a custom, portable and low-cost brain-computer interfaceplatform,”