Personalizing Smartwatch Based Activity Recognition Using Transfer Learning
PPERSONALIZING SMARTWATCH BASED ACTIVITY RECOGNITION USING TRANSFERLEARNING
Karanpreet Singh ∗ Virginia TechBlacksburg, VA [email protected]
Rajen Bhatt, Ph.D. † Qeexo Co.Pittsburgh, PA [email protected]
ABSTRACT
Smartwatches are increasingly being used to recognize hu-man daily life activities. These devices may employ differ-ent kind of machine learning (ML) solutions. One of suchML models is Gradient Boosting Machine (GBM) which hasshown an excellent performance in the literature. The GBMcan be trained on available data set before it is deployed onany device. However, this data set may not represent everykind of human behavior in real life. For example, a ML modelto detect elder and young persons running activity may givedifferent results because of differences in their activity pat-terns. This may result in decrease in the accuracy of activ-ity recognition. Therefore, a transfer learning based methodis proposed in which user-specific performance can be im-proved significantly by doing on-device calibration of GBMby just tuning its parameters without retraining its estimators.Results show that this method can significantly improve theuser-based accuracy for activity recognition.
Index Terms — activity recognition, on-device machinelearning, transfer learning, wearable devices, GBM
1. INTRODUCTION
Over the last decade, human activity monitoring devices haveseen an exceptional development. The motivation behind thisis that monitoring human activity has become increasinglyimportant because of multiple reasons. As [1] stated thatthere are many studies which link physical activity with over-all health status. Moreover, recognizing activities such aswalking, running, or cycling in patients with diabetes, obe-sity, heart disease, dementia or other mental pathologies helpsin improvement in their treatments [2].There are different kinds of heath monitoring deviceswhich have been studied in the literature such as pedometers, ∗ Ph.D. Candidate, Kevin T. Crofton Department of Aerospace and OceanEngineering, Virginia Tech, Blacksburg, VA 24060. The author performedthe work while being Machine Learning Research Intern at Qeexo Co., Pitts-burgh, PA 15206 † Director of Engineering, Machine Learning R&D, Qeexo Co., Pitts-burgh, PA 15206 smartphones, smartwatches, etc. Chan et al. [3] studied thehealth benefits of a pedometer-based physical activity inter-vention in sedentary workers. They showed improvementsin waist girth and resting heart rate in workers with increasein the number of accumulated steps per day. Smartwatchesand smartphones are other options for human activity recog-nition. These devices contain accelerometers and gyroscopesthat can help detecting one’s activity being performed basedon their body movements.In literature, many Machine Learning (ML) solutionshave been used for human activity recognition. Anguita et al. [4] studied human activity recognition on smartphones usinga multi-class Support Vector Machine (SVM). Baccouche etal. [5] presented a deep neural network to classify sequencesof human actions. Ronao and Cho [6] presented a deepconvolutional neural network (convnet) to perform efficientand effective human activity recognition using smartphonesensors.Smartwatches have some advantages over smartphones inactivity recognition. For example, Weiss et al. [7] stated thatthe smartphone could shift in a persons pocket and pocket po-sition is not ideal for tracking hand-based activities. They alsostated that smartphone-based activity recognition has limi-tations for women because they generally do not keep thephones in their pockets. However, smartwatches help in ad-dressing these issues and could perform better while makinguse of person’s hand movements.Weiss et al. [7] compared different ML algorithms us-ing smartphones and smartwatches. They trained their MLmodels on two kinds of data: 1) data from only the intendeduser (called personal models) and 2) data from every user butthe intended user (called impersonal models). They showedthat personal models perform better than impersonal modelsbut at the cost of requiring each user to provide labeled train-ing data. Shahmohammadi et al. [8] studied smartwatchbased activity recognition using active learning. They statedthat, in literature, many methods have been presented whicheven though achieve high accuracy on collected data set, butwere not personalized for the user and required large data set. a r X i v : . [ c s . L G ] S e p herefore, they used active learning to develop user person-alized models for activity recognition. These studies showthat there is need to build personalized ML models for gettinghigh accuracy for activity recognition.In this paper, we propose a novel approach to personal-ize smartwatch based activity recognition for specific user.In this approach, we tune parameters of a trained GradientBoosting Machine (GBM) without retraining its estimators.Initially, a GBM is trained on an available data set. Later, thisGBM is used and its parameters are tuned on small data froma user to improve the activity recognition of that user’s activ-ities. Only a small activity data is required from the user tore-calibrate the GBM for improving the performance of GBMfor that specific user. This technique is very beneficial for el-ders, young children and diseased patients as data collectioncould be difficult from these people for long time. The com-mercially available smartwatch tend to not to perform wellon these people. Therefore, with this motivation, the authorspresent the current novel approach to resolve this issue.
2. METHODOLOGY
In this section, we present the methodology for personaliz-ing the smartwatch based activity recognition using transferlearning. Initially, a general introduction of GBM is provided.The authors assume that the reader would have knowledgeabout methodology behind the training of estimators of GBMML model as it is not covered in this article. For those de-tails, [9] and [10] can be referred. Later, the algorithm fortuning GBM parameters for improving user-specific accuracyis presented.
The conventional ensemble algorithms like Random Forestscalculates the average of the outputs of models in the ensem-ble. However, the boosting ensemble methods are based ondifferent constructive strategy. In boosting, we add new mod-els to the ensemble sequentially during training. [10] Afterevery iteration, a new weak learner model is trained with re-spect to the error of the whole ensemble learnt so far.Friedman [9] introduced Gradient Boosting Machine(GBM) algorithm in 2001. This algorithm sequentially fitsnew machine learning models to provide a better estimate ofthe response variable. [10] The main goal of this algorithmis to build weak learners to be maximally correlated with thenegative gradient of the loss function related to the wholeensemble.
Once a GBM is trained, it is used to predict the output for anyparticular inputs. Initially, the decision functions from all theestimators is predicted. For example, the decision function, φ jp , is calculated from j th estimator for p th class. Later, afinal score is calculated for p th class in a multi-class output as f p = φ p + n (cid:88) j =1 w jp φ jp (1)where, n is the total number of estimators, φ p is the initialguess for p th class, φ jp is the decision output from j th esti-mator for p th class and w jp is the weight attached to this j th estimator.The score output for the each class is used to find the prob-ability, P ( p ) , of p th class by using softmax function as P ( p ) = exp( f p ) (cid:80) lg =1 exp( f g ) (2)where, l is the total number of classes. Finally, an outputis predicted to be the class having the maximum probability.The training of GBM could take could require significantcomputing resources and time. And, it is only possible to betrained on a limited available data set. For example, a GBMcan be trained on data based on ’running’ and ’walking’ ac-tivities available from many users. However, this data set maynot represent every kind of human behavior in real life. Thismay result in decrease in the accuracy of activity recognition.Also, training a full personalized GBM on-device is also con-sidered difficult as per current state of the art computing con-figurations.Therefore, we propose a transfer learning based approachwhere we use a GBM already trained on an available data setand later we just train the weights, w jp , on small data set froma user to improve accuracy for that user specifically. Duringthis approach, we do not alter the estimators of GBM. This ap-proach has the benefit of being possible to be done on device.For example, a trained GBM can be deployed on a smartwatchand later, a new user would be asked to perform the requiredactivities, like running and walking, while wearing the device.The new data collected from the user would be then used totrain the weights, w jp , of the GBM.To train the weights of the GBM, different loss functionscan be used. For example, if mean squared error is used asthe loss function to train the weights, the loss function can bedefined as L = 1 l l (cid:88) p =1 ( ˆ P ( p ) − P ( p )) (3)where, ˆ P ( p ) is the ground truth probability for p th class.This probability would be 1 for one of the classes and 0 forthe remaining classes for any particular input.The gradient of the loss function with respect to weight w jp can be defined as ∂L∂w jp = 2 l l (cid:88) p =1 ( ˆ P ( p ) − P ( p ))( − ∂P ( p ) ∂w jp ) (4) rain GBM using TrainingUsers Data Split Test User Data to Sets A and B Tune GBM weights on Set A Make Predictions for Set BTune GBM weights on Set B Make Predictions for Set A Return Predictions for Test Data Fig. 1 . Flowchart of Tuning of GBM Weights and Model evaluationwhere, ∂P ( p ) ∂w jp = P ( p ) ∂f p ∂w jp − ( P ( p )) ∂f p ∂w jp (5)and ∂f p ∂w jp = φ jp (6)The gradient of the loss function can be used to update theweights as follows, ˆ w jp = w jp − η ∂L∂w jp (7)In the current work, stochastic gradient descent is usedby making small batches of the small user data set to updatethe weights of the GBM. Therefore, batch size and learningrate are the hyper-parameters that can be tuned to improveaccuracy on the validation set.
3. APPLICATION AND RESULTS
In this section, the methodology explained in Section 2 is ap-plied on two publicly available data sets. The first data set isfrom [11, 12] and is called ”Daily and Sports Activities DataSet”. The second data set is from [13] and is called ”PAMAP2Data-set: Physical Activity Monitoring”.In the current work, one-user-out cross-validation (CV)and F-1 scores are compared on these data sets for baselineand tuned GBM. Fig. 1 shows the flowchart of the one-user-out CV procedure for tuning the GBM weights and modelevaluation. The baseline GBM one-user-out CV is calculatedby training GBM on ( N − users and tested on N th user. Fortuning the GBM weights, the N th user’s data is split to SetsA and B. Initially, GBM weights are tuned on Set A and then,the tuned GBM is used to make predictions for Set B and viceversa. This way, tuned GBM is used to calculate one-user-outCV. A part of the tuning data is used as a validation set tochoose the final model based on validation set accuracy. This data set has 19 different activities performed by 8 dif-ferent subjects. The data is collected using accelerometer,gyroscopes, and magnetometers attached at different parts ofbody of the subjects. To show the potential of the presentedapproach in this paper, we selected four different activities:Running, Biking, Resting and Walking from this data set. Outof all the sensors, we only used the data from accelerometersensor attached to arms/wrists.
The data was collected at 25 Hz of sampling frequency. Inthe current work, 1 sec of latency is considered. This meansthat 25 samples, collected every second, are used to gener-ate one instance of features. Tab. 1 shows different featurescomputed using these samples. These features are calculatedusing accelerometer data about x , y and z axes. Table 1 . List of Features Used in Activity Classfication
Features
MeanStandard DeviationSkewnessAuto-correlationRangeRoot Mean Square
Initially, for comparison, the baseline GBM model one-user-out CV accuracy is compared with other ML models as shownin Fig. 2. It is seen that for ’Rest’ and ’Run’ classes, everymodel has high accuracy. However, the GBM perform bet-ter than other models for ’Bike’ and ’Walk’ classes. This isalso due to the reason that different users could walk and docycling differently than other users. ubject T r u e P o s i t i v e R a t e T r u e P o s i t i v e R a t e T r u e P o s i t i v e R a t e NaiveBayes Decision Tree SVM LogisticRegression GBM
Baseline GBM Tuned GBM
Fig. 2 . Comparison of accuracy of GBM with other machinelearning models on data set ILater on, the tuning algorithm is used on this data set toimprove one-user-out CV accuracy. Fig. 3 shows average in-crease in the accuracy of each subject in data set I. It can beseen that there is some increase in the accuracy of every sub-ject. There is a significant increase in the accuracy of ’Bike’and ’Walk’ class for subject
This data set has 12 different activities performed by 9 differ-ent subjects. Out of many different sensors used, we extractedonly accelerometer data attached at the subject’s wrist. Threedifferent activities: Biking, Resting and Walking are consid-ered in this study. We only used 7 subjects’ data due to miss-ing data of 2 subjects for the activities of our interest.
Table 2 . Comparison of F1 scores of baseline and tuned GBMacross subjects of data set I
Subject Baseline Tuned GBM StandardDeviation
This data set was collected at 100 Hz. As mentioned in abovesection, we again considered 1 sec of latency, meaning, weused 100 samples are used to calculate features instances.Similar features are calculated as mentioned in Section 3.1.1.
In this section, the baseline GBM trained on data set II is usedto tune its weights for improving the performance. Again, theflowchart in Fig. 1 is used to find on-user-out CV accuracy.Fig. 6 shows average increase in the one-user-out CV accu-racy of each class of subjects and overall increase in the CVaccuracy by tuning the GBM.Tab. 3 shows the comparison of F-1 scores for baselineand tuned GBM. The overall F-1 score increase from 0.9307to 0.9619. There is a significant increase in the subject F-1score of
4. CONCLUSION
In this paper, we propose a novel approach to personalizesmartwatch based activity recognition for a specific user. Thealgorithm only tune the weights of a trained GBM without re-training its estimators. The approach has the benefit of beingpossible to be done on device. Only a small amount of ac-tivity data is required from the user to re-calibrate the GBMfor improving the performance of GBM for that specific user.The approach has shown significant increase in the accuracyand F-1 scores for the users who have low accuracy for certainactivity classes. efore Tuning After TuningSubject
Fig. 3 . Average increase in the accuracy of subjects in Data Set 1
Subject T r u e P o s i t i v e R a t e T r u e P o s i t i v e R a t e T r u e P o s i t i v e R a t e NaiveBayes Decision Tree SVM LogisticRegression GBM
Baseline GBM Tuned GBM
Fig. 4 . Overall accuracy of tuned GBM on data set 1In future, the approach can be applied on other activitiesor gesture recognition for creating user personalized machinelearning models.
5. REFERENCES [1] Emil Jovanov, Aleksandar Milenkovic, Chris Otto, PietDe Groen, B Johnson, S Warren, and G Taibi, “AWBAN system for ambulatory monitoring of physicalactivity and health status: applications and challenges,”in . IEEE, 2006, pp. 3810–3813.[2] Oscar D Lara and Miguel A Labrador, “A survey on hu-man activity recognition using wearable sensors,”
IEEE
Table 3 . Comparison of F1 scores of baseline and tuned GBMacross subjects of data set II
Subject Baseline Tuned GBM StandardDeviation communications surveys & tutorials , vol. 15, no. 3, pp.1192–1209, 2013.[3] Catherine B Chan, Daniel AJ Ryan, and Catrine Tudor-Locke, “Health benefits of a pedometer-based physicalactivity intervention in sedentary workers,”
Preventivemedicine , vol. 39, no. 6, pp. 1215–1222, 2004.[4] Davide Anguita, Alessandro Ghio, Luca Oneto, XavierParra, and Jorge L Reyes-Ortiz, “Human activity recog-nition on smartphones using a multiclass hardware-friendly support vector machine,” in
International work-shop on ambient assisted living . Springer, 2012, pp.216–223.[5] Moez Baccouche, Franck Mamalet, Christian Wolf,Christophe Garcia, and Atilla Baskurt, “Sequential deeplearning for human action recognition,” in
Internationalworkshop on human behavior understanding . Springer,2011, pp. 29–39. ubject T r u e P o s i t i v e R a t e T r u e P o s i t i v e R a t e T r u e P o s i t i v e R a t e NaiveBayes Decision Tree SVM LogisticRegression GBM
Baseline GBM Tuned GBM
Fig. 5 . Comparison of ROC curves of subject
Before Tuning After TuningSubject
Fig. 6 . Average increase in the accuracy of subjects in data set II[6] Charissa Ann Ronao and Sung-Bae Cho, “Human ac-tivity recognition with smartphone sensors using deeplearning neural networks,”
Expert systems with applica-tions , vol. 59, pp. 235–244, 2016.[7] Gary M Weiss, Jessica L Timko, Catherine M Gallagher,Kenichi Yoneda, and Andrew J Schreiber, “Smartwatch-based activity recognition: A machine learning ap-proach,” in . IEEE,2016, pp. 426–429.[8] Farhad Shahmohammadi, Anahita Hosseini, Christine EKing, and Majid Sarrafzadeh, “Smartwatch based ac-tivity recognition using active learning,” in
Proceed-ings of the Second IEEE/ACM International Conferenceon Connected Health: Applications, Systems and Engi-neering Technologies . IEEE Press, 2017, pp. 321–329.[9] Jerome H Friedman, “Greedy function approximation:a gradient boosting machine,”
Annals of statistics , pp.1189–1232, 2001. [10] Alexey Natekin and Alois Knoll, “Gradient boostingmachines, a tutorial,”
Frontiers in neurorobotics , vol. 7,pp. 21, 2013.[11] Kerem Altun, Billur Barshan, and Orkun Tunc¸el, “Com-parative study on classifying human activities withminiature inertial and magnetic sensors,”
Pattern Recog-nition , vol. 43, no. 10, pp. 3605–3620, 2010.[12] Billur Barshan and Murat Cihan Y¨uksek, “Recognizingdaily and sports activities in two open source machinelearning environments using body-worn sensor units,”
The Computer Journal , vol. 57, no. 11, pp. 1649–1667,2014.[13] Attila Reiss and Didier Stricker, “Introducing a newbenchmarked dataset for activity monitoring,” in201216th International Symposium on Wearable Computers