Analysis of Contraction Effort Level in EMG-Based Gesture Recognition Using Hyperdimensional Computing
Ali Moin, Andy Zhou, Simone Benatti, Abbas Rahimi, Luca Benini, Jan M. Rabaey
PPublished as a conference paper at the IEEE BioCAS 2019
Analysis of Contraction Effort Level in EMG-Based GestureRecognition Using Hyperdimensional Computing
Ali Moin ∗ , Andy Zhou ∗ , Simone Benatti † , Abbas Rahimi ∗‡ , Luca Benini †‡ , Jan M. Rabaey ∗∗ Berkeley Wireless Research Center, EECS Department, University of California, Berkeley. † DEI, University of Bologna, Italy. ‡ Integrated System Laboratory, ETH Zurich, Switzerland.Corresponding Author Email: [email protected]
Abstract —Varying contraction levels of muscles is a big chal-lenge in electromyography-based gesture recognition. Some usecases require the classifier to be robust against varying forcechanges, while others demand to distinguish between differenteffort levels of performing the same gesture. We use brain-inspired hyperdimensional computing paradigm to build clas-sification models that are both robust to these variations andable to recognize multiple contraction levels. Experimental resultson 5 subjects performing 9 gestures with 3 effort levels showup to 39.17% accuracy drop when training and testing acrossdifferent effort levels, with up to 30.35% recovery after applyingour algorithm.
I. I
NTRODUCTION
Hand gestures are an integral part of human communicationas well as object manipulation and dexterity. Electromyogra-phy (EMG)-based pattern recognition has shown great poten-tial in classifying hand gestures, where EMG features gatheredfrom the sensors on the skin serve as inputs to machinelearning algorithms. Although being non-invasive makes itan attractive method, it is highly prone to signal variationscaused by factors such as changing limb position [1], electrodeshift [2], and force change [3]. While the first two areundesired phenomena that the classifier has to ideally be robustagainst, the last could occasionally be desired in applicationssuch as proportional control of prosthetic hands.A subject can exert different levels of effort while perform-ing a gesture, resulting in different EMG signal properties.Scheme and Englehart [4] have shown up to 50% error ratewhen the classifier was trained and tested at different force lev-els from 20% to 80% maximal voluntary contraction (MVC),compared to moderate 7% to 19% error rate when trained andtested at the same level. Previous works have suggested to pickspecific force levels that yield minimum accuracy degradationacross all force levels as training dataset and to extract featuresthat are more invariant against contraction levels as the inputto the classifier [5], [6].In this paper, we propose building a general classificationmodel based on hyperdimensional (HD) computing [7] to dealwith varying muscle contraction effort levels. HD computinghas shown promising results in classification tasks usingbiosignals such as EMG in recognizing hand gestures [8] andelectrocorticography (ECoG) for seizure detection with one-shot learning [9]. With slight modifications to our previouslyintroduced encoding scheme [8], we analyze the muscle con-traction level variations in two different ways, depending on
Fig. 1. Hand gesture classes used in the study. The single degree-of-freedom(DOF) gesture subset includes individual finger flexions and extensions. Themulti-DOF gesture subset includes isometric hand postures involving multiplefingers. the application: If discrimination between different gestures isthe only goal, the classifier should output the same gestureclass regardless of the subject’s effort level. If, on the otherhand, different effort levels are relevant to the application(e.g. controlling different levels of force for gripping usinga prosthetic hand), different effort levels for the same gesturemust be treated as separate output classes. A classifier based onHD computing can be naturally used in both of these scenarios.In the former case, it can include minimum amount of trainingdata from multiple effort levels for training each gesture tobuild an inclusive model that ignores effort level variations.In the latter, distinguishing between different effort levels ofthe same gesture translates to simply defining a separate classfor each level of contraction. A dataset of 5 human subjectsperforming 9 hand gestures (Fig. 1) with low, medium, andhigh contraction effort levels was recorded using a wireless,high-channel count EMG recording system [8] which providedvisual feedback of effort level. Classification accuracy resultsfor both gesture-only and gesture+effort cases are presented.II. E
XPERIMENT S ETUP
We used a custom, wireless 64-channel EMG signal acqui-sition device [8] to record a dataset of EMG signals from a r X i v : . [ c s . H C ] A ug ig. 2. Visual feedback of real-time contraction effort level to the user. Thebar represents the mean signal energy across all channels as a measure ofcontraction effort. The users are asked to reach 25%, 50%, and 75% of theirmaximum voluntary contraction (MVC). During calibration, a multiplier andan offset are determined such that the rest state and the MVC map to 0 and100, respectively. five able-bodied, adult male subjects . A flexible 16x4 arrayof electrodes was wrapped completely around the subject’supper forearm, capturing activity of the extrinsic flexor andextensor muscles involved in finger movements with / s sampling rate. A single Ag/AgCl electrode is attached to theelbow to provide a reference voltage for all channels. The rawrecorded signals were wirelessly transmitted to a base stationfor offline processing. Additionally, we calculated the meansignal energy across all channels as a measure of contractioneffort level, and illustrated the value as a bar graph (Fig. 2)in the graphical user interface (GUI). This served as a visualfeedback to the subject in real-time.For this study, we chose a set of gestures consisting ofmovements of the thumb, index, and middle fingers to modelsimple grasping actions (Fig. 1): index finger flexion andextension, middle finger flexion and extension, and thumbflexion and extension as single degree-of-freedom (DOF)gesture subset, and one, two, and fist as multi-DOF subset.For each gesture, we started with a calibration phase duringwhich the subject was asked to perform the gesture withthe maximum contraction effort, also known as maximumvoluntary contraction (MVC). This value was normalized tomap to 100% in the GUI feedback bar graph (Fig. 2). Thesubject was asked to target three different effort levels (loweffort at 25%, medium effort at 50%, and high effort at 75%)for each gesture, repeating each 5 times.Each trial lasted 8 seconds (Fig. 3), with 3 seconds ofrest before the next trial. The subject was told to begin thegesture within a 2-second transition window which wouldcontain the transient, non-stationary part of the EMG signal forthat gesture. After the 2-second transition window, the subjectwas asked to hold the gesture for 4 seconds, constituting thesteady-state part of the EMG signal. Finally, the subject wasdirected to return to the rest position within another 2-secondtransition window. These directions ensured that the steady-state portion of the gesture could easily be labeled as part ofthe middle 4 second segment. Data were automatically labeledwith the gesture class and saved as .mat files for processingin MATLAB (MathWorks, Inc.). Dataset and scripts available at https://github.com/flexemg/flexemg v2 Fig. 3. Representative EMG signals from one electrode channel recordedduring a low, medium, and high effort level trial of the same gesture. Thevertical dotted lines divide a single gesture trial into transition periodsand a hold period based on the instructions given to the subject. The colorof the waveform indicates the effort level, as measured by windowed signalpower (RMS calculated over
200 ms windows with
150 ms overlap).
All experiments were performed in strict compliance withthe guidelines of IRB and were approved by the Committeefor Protection of Human Subjects at University of California,Berkeley (Protocol title: Flex EMG Study. Protocol number:2017-10-10425).III. C
LASSIFICATION A LGORITHM
A. HD Computing Background
HD computing employs hypervectors with very high dimen-sionality (e.g. 10,000) to represent information, analogous tothe way the human brain utilizes vast circuits of billions ofneurons and synapses [7]. In general, a fixed symbol table, oritem memory (IM), is built from an initial set of HD hyper-vectors taken randomly from a high-dimensional (e.g. 10,000-dimensional) space. Each hypervector consists of an equalnumber of randomly placed +1 ’s and − ’s. A fundamentalproperty is that, with a very high probability, hypervectorswithin a randomly generated IM will all be orthogonal to eachother, i.e. any pair of hypervectors will differ by approximately5,000 bits. These hypervectors can be combined to form newcomposite HD hypervectors using well-defined vector spaceoperations, including point-wise multiplication ( ∗ ), point-wiseaddition ( + ), scalar multiplication ( × ), and permutation ( ρ ).Because of the high dimensionality and randomness, HDhypervectors can be combined while preserving the originalinformation.Fig. 4 summarizes the process of encoding raw EMG datainto HD hypervectors for training and inference. Data is firstpreprocessed to extract the features to be used as inputs to theHD algorithm. We used mean absolute value (MAV) with non-overlapping windows of 50 samples as input features. Featuresare then encoded spatially (across 64 channels) and temporally(
250 ms windows) into HD hypervectors exactly as describedin [8]. Spatiotemporal hypervectors calculated using data from ig. 4. High-level flow diagram for encoding 64 electrode channels ofEMG data into hypervectors and outputting the classified gesture label usinghyperdimensional (HD) computing algorithm. each gesture class are bundled together (i.e., summed) andbipolarized (i.e. positive elements replaced by +1 and negativeelements replaced by − ) to form a binary prototype hyper-vector representing that class. During training phase, theseprototype hypervectors are stored in the associative memory(AM) with their corresponding labels. During inference, thetest hypervector is compared to each entry of the AM usingcosine similarity as the distance metric. The inferred gestureis selected by finding the closest prototype hypervector in theAM. B. HD Model for Contraction Effort Levels
In contrast to many state-of-the-art classification algorithmsthat often require a big training dataset, HD computingachieves high classification accuracies with small amounts oftraining data, i.e. only 1 out of 5 trials of each gesture in ourcase. Therefore, building inclusive prototype hypervectors thatcontain data from multiple effort levels is fast. If, on the otherhand, data from individual effort levels is used to form theprototype hypervectors, HD model will distinguish the effortlevel in addition to the gesture itself.
1) Gesture-Only Classification:
If the only goal is to dis-criminate between different gestures regardless of the subject’seffort level, a single gesture prototype hypervector can beformed to include information from those different effortcontexts. This can be done by accumulating spatiotemporalhypervectors from multiple effort levels, and saving its bipo-larized hypervector in the AM. If the prototype hypervectorsof the two effort levels are already calculated and bipolarized,however, another approach is to merge them into a singleprototype hypervector by randomly taking 5000 elements (halfof the elements) from each prototype hypervector.
2) Gesture+Effort Classification:
If discriminating be-tween different effort levels of gestures is desired, each { gesture,effort } pair must be treated as a separate output class.In this case, prototype hypervectors for each effort level canbe added to the model as new entries in AM.Note that a potential third case could involve adding newprototype hypervectors for each effort level to the AM whilepreserving the number of gesture classes, allowing multipleprototype entries to represent the same class. While this willimprove the classification accuracy comparing to the case where prototype hypervectors were merged, it costs morememory and computation resources as three prototype hyper-vectors have to be generated and stored for each gesture class.IV. R ESULTS
We first treated different effort levels as different contextsof the same gesture class. An initial model was trained withgestures from one effort level context. It was then crossvalidated (training with one trial, inference with remainingfour trials) within the same effort level and also used toclassify gestures from the other level within the pair, withoutmerging the models (Fig. 5(a-c), first and second pairs). Whentraining and testing within the same effort level context,classification accuracy remained better than 93.11%. However,across different effort levels, classification accuracy droppedby between 16.57% and 39.17%, with the worst performancewhen the difference between effort levels was highest, i.e. lowand high effort. After merging the prototype hypervectors toinclude both effort contexts, the classification accuracy was re-covered to above 78.21% (Fig. 5(a-c), third pairs). The poorestrecovered accuracies resulted from training an initial modelon medium or high effort level gestures, and then mergingwith low effort gestures. While prototype hypervectors formedium and high effort level gestures were more similar toeach other, prototype hypervectors for low effort level gestureswere more distant due to a smaller variance in the calculatedfeature values.For a model trained with all three contexts by accumulat-ing their spatiotemporal hypervectors before bipolarization,accuracy was at least 88.19% for all three effort contexts(Fig. 5(d)). In this case, the all-inclusive final hypervector isweighted to be more similar to medium and high effort levelprototype hypervectors enabling higher accuracies in thosecontexts.When treating different effort levels of a single gesture asdifferent classes, we trained a new AM entry for each gestureand effort level, increasing the total number of classes. Wecalculated classification accuracy in two different ways: Forthe first method (Fig. 6, red bars), an accurate classificationrequired matching both the gesture type and its effort levelto the label. For the second (Fig. 6, purple bars), an accurateclassification required only matching the gesture type. Notably,if we disregard the effort level classification output from thismodel, we achieve a better gesture-only classification accuracythan in the case where we treated different effort levels asdifferent contexts. V. C
ONCLUSION
We have presented methods based on HD computingparadigm that address some of the challenges caused by vari-ous muscle contraction levels in EMG-based gesture recogni-tion. Our experimental data showed significant classificationaccuracy degradation when training and testing across differenteffort levels. We demonstrated that high accuracy can besimply recovered using a minimum amount of data (only asingle trial) from each effort level. Moreover, we verified ig. 5. Classification accuracy measured before and after merging the models when treating effort levels as different contexts of the same gesture class.Accuracies across effort contexts before and after merging were calculated for each pair of effort levels: low (L) and medium (M) in (a), low and high (H)in (b), and medium and high in (c). Accuracy for each effort level was also calculated using a model trained with all three effort level contexts (d).Fig. 6. Classification accuracy measured when treating different effort levelsof the same gesture as different classes. Accuracy was calculated as thesuccess rate of matching both gesture type and effort level (red) as well asgesture type only (purple). that the HD model is capable of including new classes todistinguish among multiple effort levels of gestures withoutthe need to change the existing model.A
CKNOWLEDGMENT
The authors would like to thank Alisha Menon, GeorgeAlexandrov, Senam Tamakloe, Jonathan Ting, Natasha Ya-mamoto, Yasser Khan, Fred Burghardt, Ken Lutz, Haylie Wu,Profs. Elad Alon, Rikky Muller, Ana C. Arias, NovacentrixCorp. and Cortera Neurotechnologies Inc. This work wassupported in part by the CONIX Research Center, one ofsix centers in JUMP, a Semiconductor Research Corporation(SRC) program sponsored by DARPA. Support was also received from sponsors of Berkeley Wireless Research Centerand Savio computational cluster resource provided by theBerkeley Research Computing program at UC Berkeley.R
EFERENCES[1] Y. Geng, P. Zhou, and G. Li, “Toward attenuating the impact of arm posi-tions on electromyography pattern-recognition based motion classificationin transradial amputees,”
Journal of neuroengineering and rehabilitation ,vol. 9, no. 1, p. 74, 2012.[2] A. J. Young, L. J. Hargrove, and T. A. Kuiken, “Improving myo-electric pattern recognition robustness to electrode shift by changinginterelectrode distance and electrode configuration,”
IEEE Transactionson Biomedical Engineering , vol. 59, no. 3, pp. 645–652, 2011.[3] A. H. Al-Timemy, R. N. Khushaba, G. Bugmann, and J. Escudero,“Improving the performance against force variation of EMG controlledmultifunctional upper-limb prostheses for transradial amputees,”
IEEETrans. on Neural Systems and Rehab. Eng. , vol. 24, no. 6, 2015.[4] E. Scheme and K. Englehart, “Electromyogram pattern recognition forcontrol of powered upper-limb prostheses: state of the art and challengesfor clinical use.”
Journal of Rehabilitation Research & Development ,vol. 48, no. 6, 2011.[5] R. N. Khushaba, A. Al-Timemy, S. Kodagoda, and K. Nazarpour,“Combined influence of forearm orientation and muscular contraction onEMG pattern recognition,”
Expert Systems with Applications , vol. 61, pp.154–161, 2016.[6] J. He, D. Zhang, X. Sheng, S. Li, and X. Zhu, “Invariant surface EMGfeature against varying contraction level for myoelectric control based onmuscle coordination,”
IEEE journal of biomedical and health informatics ,vol. 19, no. 3, pp. 874–882, 2014.[7] P. Kanerva, “Hyperdimensional computing: An introduction to computingin distributed representation with high-dimensional random vectors,”
Cognitive computation , vol. 1, no. 2, pp. 139–159, 2009.[8] A. Moin, A. Zhou, A. Rahimi, S. Benatti, A. Menon, S. Tamakloe,J. Ting, N. Yamamoto, Y. Khan, F. Burghardt et al. , “An EMG gesturerecognition system with flexible high-density sensors and brain-inspiredhigh-dimensional classifier,” in , 2018.[9] A. Burrello, K. Schindler, L. Benini, and A. Rahimi, “One-shot learningfor iEEG seizure detection using end-to-end binary operations: Lo-cal binary patterns with hyperdimensional computing,” in2018 IEEEBiomedical Circuits and Systems Conference (BioCAS)