EmoconLite: Bridging the Gap Between Emotiv and Play for Children With Severe Disabilities
Javad Rahimipour Anaraki, Chelsea Anne Rauh, Jason Leung, Tom Chau
NNoname manuscript No. (will be inserted by the editor)
EmoconLite: Bridging the Gap Between Emotiv and Playfor Children With Severe Disabilities
Javad Rahimipour Anaraki, Chelsea AnneRauh, Jason Leung, Tom Chau
Received: date / Accepted: date
Abstract
Brain-computer interfaces (BCIs) allow users to control computer appli-cations by modulating their brain activity. Since BCIs rely solely on brain activity,they have a lot of potential as an alternative access method for engaging childrenwith severe disabilities and/or medical complexities in Therapeutic Recreationand leisure. In particular, one commercially available BCI platform is the EmotivEPOC headset, which is a portable and affordable electroencephalography (EEG)device. Combined with the EmotivBCI software, the Emotiv system can generatea model to discern between different mental tasks based on the user’s EEG sig-nals in real-time. While the Emotiv system shows promise for use by the pediatricpopulation in the setting of a BCI clinic, it lacks integrated support that allowsusers to directly control computer applications using the generated classificationoutput. To achieve this, users would have to create their own program, which canbe challenging for those who may not be technologically inclined. To address thisgap, we present a free and user-friendly BCI software application called Emocon-Lite. Using the classification output from EmotivBCI, EmoconLite allows users toplay YouTube video clips and a variety of video games from multiple platforms,ultimately creating an end-to-end solution for users. With its use in the HollandBloorview Kids Rehabilitation Hospital’s BCI clinic, EmoconLite will bridge thegap between research and clinical practice, providing children with access to BCItechnology and supporting BCI-enabled play.
Keywords
Brain-computer interface · Electroencephalography · Emotiv headset · Assistive technology · Therapeutic recreation
J. R. Anaraki, J. Leung and T. ChauInstitute of Biomedical Engineering, University of Toronto, Toronto, ON, CanadaE-mail: [email protected], [email protected], & [email protected]. A. Rauh and T. ChauHolland Bloorview Kids Rehabilitation Hospital, Toronto, ON, CanadaE-mail: [email protected] & [email protected] a r X i v : . [ c s . H C ] F e b Javad Rahimipour Anaraki, Chelsea Anne Rauh, Jason Leung, Tom Chau
Brain-computer interfaces (BCIs) allow users to interact with computers usingonly their brain activity. By performing mental tasks that generate specific brainactivity patterns, users can use BCIs to control various applications and devices(Mak and Wolpaw, 2009). When used by children with disabilities and/or medicalcomplexities in the context of play and leisure, BCIs may open up a world of pos-sibilities by reducing leisure disparities and supporting active, independent andengaged participation in leisure and life. However, one of the barriers preventingthe widespread use of BCIs is their accessibility outside of the research environ-ment. To this end, a program that facilitates access to BCI at home and in theclinic can bridge the gap between research and clinical practice.Through collaboration with children with disabilities and their families, a BCIclinic can support the well-being and quality of life of the pediatric populationon an ethically sustainable research foundation, ultimately contributing to theadvancement of BCI technology and further benefit for its users (Mietola et al.,2017). Besides, such a program can also facilitate an environment of fun, function,fitness, family, friendship and play that forms the foundation of sustained skilldevelopment and a holistic perspective to healthcare (Rosenbaum and Gorter,2012). As one of the founding partners of BCI Canada collaborative network (BCI-CAN), Holland Bloorview Kids Rehabilitation Hospital has taken on this missionthrough its BCI clinic, which introduces children with severe disabilities, and theirfamilies to BCIs (Kinney-Lang et al., 2020).One of the factors that dictate the success of the BCI clinic is the accessibilityof BCIs to its users. Considering that readily available hardware and softwareis more likely to reach a larger number of potential users, the Emotiv EPOCxheadset was chosen for the BCI clinic since it is a commercially available, wirelessBCI headset based on electroencephalography (EEG) technology. With 14 saline-based electrodes that eliminate sticky gel’s inconvenience, the headset is easy toset up and is therefore ideal for use in home and clinical applications. Using theEmotiv EPOC + headset (previous iteration of Emotiv EPOCx), Zhang et al.determined that typically developing children between the ages of 6 and 18 “canquickly achieve control and execute multiple tasks using simple EEG-based BCIsystems” mandating exploration for children with disabilities and complex medicalneeds (Zhang et al., 2019).Another essential component of the BCI clinic is a software system that sup-ports interactions between the hardware, the user interface, and other applications.The Emotiv system supports toolboxes such as NodeRed that allow users to in-terface Emotiv with external applications for BCI integration. However, for userswho do not know how to program, it may be challenging to create and maintainsuch an interface.In this paper, we present EmoconLite - a free BCI software system developed for the BCI clinic that can be used with the Emotiv hardware. The EmoconLiteapplication provides a simple interface that allows users to access media. It alsoallows users to play BCI-compatible games, both locally and online. In section2, we explain EmoconLite’s software architecture and functionality. We discuss http://hollandbloorview.ca/emocon moconlite 3 UserEmoconLiteCortex APIEmotiv headset Activities – YouTube – HelpKidzLearn – Brain Joust – Steam
Fig. 1
EmoconLite architecture the role of EmoconLite within the Holland Bloorview BCI clinic in section 3, andconclude our paper in section 4.
EmoconLite is a software application designed to bridge the gap between theEmotivBCI system and other computer programs (see Figure 1). The applicationis implemented in Python 3.7 (Python 3.7.0) using PySide2 (PySide2). The app iscross-platform and can be run on Windows, Linux and macOS with small adjust-ments. However, since EmotivBCI is only compatible with Windows and macOS,we mainly focused on these operating systems. The EmoconLite graphical userinterface (GUI) is divided into five sections based on the user workflow:
Setup andtraining , Activity , Accuracy Sensitivity , Controls , and
Status .2.1 User WorkflowThe workflow of EmoconLite is depicted in Figure 2.
The
Setup and training section allows users to train the EmotivBCI system (Emo-tivBCI) to recognize different mental states. In addition to the neutral mentalstate, there are 13 mental tasks users can configure, allowing up to 13 different in-puts to be provided. The current version of EmoconLite accepts binary tasks (e.g.neutral vs any mental task), and translates them into “on” and “off” commands.However, a future iteration of the application will handle multiple tasks and mapthem to arrow keys to provide more degrees of control. In this section, users cancreate a new profile, train the BCI model to recognize their chosen mental tasks,and configure the profile to link mental tasks to different commands.
In the
Activity section, the configured profiles can be used for a list of four ac-tivities: to launch a YouTube video clip and toggle the play and pause functions,
Javad Rahimipour Anaraki, Chelsea Anne Rauh, Jason Leung, Tom Chau
Launch EmoconLiteConfigure user profileChoose an activityAdjust accuracy sen-sitivity thresholdPress
Start to initiateconnection to Cortex APIReceived class labels Check classlabels for thelast 10 samplesSend keystrokesto target activity No keystrokes sentPress
Stop toend connection | positive | > = threshold | p o s i t i v e | < t h r e s h o l d Fig. 2
User workflow diagram, where | positive | is the number of positive labels, and threshold is the chosen value for the sensitivity to play HelpKidzLearn games (HelpKidzLearn), to play the Brain Joust game(Brain Joust), or to play Steam games (Steam) with their friends. Clicking onthe YouTube button will open up a new browser window, where users can selecttheir desired YouTube video clip. The user can then perform a mental task (i.e.any task other than neutral) to play or pause the video clip. Users can also ac-cess HelpKidsLearn, an online learning platform for children of all abilities withaccessible games and activities. Another activity available to users is Brain Joust- an accessible video game developed by BCI Games (BCI Games). Finally, Emo-conLite also supports a variety of games available on the video game distributionservice, Steam. By leveraging Steam’s Remote Play functionality, users can playonline and multiplayer games with their friends over the internet. Users can toggle an interactive slider to adjust the accuracy sensitivity thresholdof EmoconLite’s classification algorithm, which uses Emotiv’s internal classifiermodel to classify EEG samples as neutral or as one of the configured mental tasks.Using a moving window approach, the class of the current sample is determinedbased on the classification results of a window of samples. For instance, a windowof the 10 last classification results is used, corresponding to approximately the last second of the EEG signal. If the number of samples classified as a particular classwithin the window is higher than the sensitivity threshold, the current sample isconsidered that class. Users can increase the sensitivity threshold if there are toomany false activations and decrease the sensitivity threshold if there are too manymissed activations. moconlite 5
When the user wants to use EmoconLite to send commands to their chosen activity,they can press the
Start button. Emocon will then initiate a background threadto communicate with the Emotiv headset through a WebSocket using the EmotivApplication Programming Interface (API) called Cortex API (Cortex API). Aftermaking sure the headset is connected and the proper user profile is selected, thethread will start a session to receive a stream of classification labels from theCortex API and simulate a keystroke press to the selected application. When theuser is done with the activity, they can press the
Stop button to disconnect theEmotiv to stop it from providing further input.
All the messages related to the initiation and execution of commands are shownin the status bar. If any error occurs while connecting to the Emotiv headset andCortex API, a corresponding message will appear in the status bar to help userstroubleshoot the problem. During an activity, the number of positive (i.e. mentaltask) and negative (i.e. neutral) class labels over the last 10 samples will also beshown in the status bar. This makes it easier for the user to adjust the accuracysensitivity threshold.2.2 Data structureData streamed to and from the Cortex API are in the JSON format. The user’smental tasks can be accessed by subscribing to the Cortex API. EmoconLite con-tinuously parses the JSON output object from the Cortex API and checks if thedesired mental task is performed or not, based on the model created for each userinternally in the EmotivBCI application. The appropriate keystroke simulationwill be sent to the target activity based on this output.2.3 ScreenshotFigure 3 shows the GUI of EmoconLite. The GUI layout follows each step in theuser workflow to make it more intuitive.2.4 DocumentationA quick guide documentation has been created to walk users through the step-by-step process to download and setup EmoconLite, along with the supportedactivities such as Brain Joust and Steam. This setup prepares EmoconLite to communicate with EmotivBCI on the computer. The quick guide is an essentialreference for family and clinicians to provide set up and training for children.The quick guide is made available in Arabic, Dutch, French, Chinese, Persian,Portuguese, and Spanish to reduce language barriers. http://hollandbloorview.ca/emocon Javad Rahimipour Anaraki, Chelsea Anne Rauh, Jason Leung, Tom Chau
Fig. 3
EmoconLite app
The inherent nature of a child is to play and have fun. This instinctive ability andhuman right to play allows for childhood discovery by observation and explorationof the world around them, supporting growth, learning, and development. Active,engaged and independent play participation within leisure and recreation experi-ences has many benefits to childhood development. Based on clinical observationand evidence-based practice, there is an identified need to support children withsevere disabilities and/or medical complexities in their active and independentengagement in a Therapeutic Recreation BCI program via assisted technology(Ramella et al., 2009). Rather than focusing on switches that require physicalmovement that may remain elusive, a BCI focuses on its user’s internal capacity,utilizing their brain signals to mobilize their capabilities to engage and self-initiateparticipation actively. BCI, as an access method, activates the users’ strengths andabilities, to enhance capabilities.Wise described capability as “the freedom to choose from a set of opportuni-ties related to what one wants to do and be” (Wise, 2018). The real freedom topursue, achieve and participate within activities of life that are deeply valued. Acapability- and strength-based approach is imperative to support the human dig-nity of children with disabilities and is essential to their well-being (Sayer, 2011;Wise, 2018; Mietola et al., 2017; Hood and Carruthers, 2016). A BCI approach re-quires co-creation and collaboration between clients, families, clinicians, and theirsupporting partners to support dignity, capabilities, health, and the well-being ofchildren with disabilities. All of these stakeholders will require access to the tech- nology, training with the technology, co-creation of participatory experience, andparticipatory action research and action sensitive research within a psychologicallysafe environment.As Huber and colleagues proposed, health is the “ability to adapt and selfmanage in the face of social, physical, and emotional challenges” (Huber et al., moconlite 7
With the creation of the EmoconLite application, Emotiv technology becomesmore practically accessible. In its application within the BCI clinic, EmoconLitecan better support play and skill development for children with severe disabilitiesand/or medical complexities during leisure. To the best of our knowledge, this isthe first effort to design, implement and provide a free application to the users
Javad Rahimipour Anaraki, Chelsea Anne Rauh, Jason Leung, Tom Chau who are interested in utilizing Emotiv headsets to perform a task within an inter-disciplinary training environment and a Therapeutic Recreation BCI program.Due to the world’s current status, the pandemic has created unique challengesrequiring creative solutions to healthcare. As the world has moved to online for-mats, such as Zoom and Ontario Telemedicine Network, so has the BCI clinic. ABCI-at-home virtual program is the devised strategy across the BCI-CAN Networkto support client and family continued access and engagement during the currentCOVID-19 pandemic (Kinney-Lang et al., 2020). One of the BCI clinic’s aims isto ensure the integration of the BCI systems within the community, especially athome where experiences can be shared with family and friends. These experiencescan also encourage the user, further motivating continued engagement.One of the notable limitations of EmoconLite is the number of mental tasksit supports. Since the current version of the software only supports binary clas-sification to distinguish between the neutral state and mental tasks, this limitsthe number of inputs users can provide to external applications. Another limita-tion of EmoconLite is that it builds upon the internal classification algorithm ofthe EmotivBCI software. Since this is a proprietary algorithm, the lack of trans-parency makes it difficult to understand the underlying classification approach orto potentially extend it. Future iterations of the application would expand thesupport for more mental tasks to give users more degrees of control and imple-ment open-source classification algorithms to allow developers to improve on theclassification performance. It would also be beneficial to expand the repositoryof supported activities and incorporate accessibility features in the GUI of theEmoconLite application. In addition, future works will also look into extendingEmoconLite to control a variety of effector devices, such as Bluetooth toys anddevices. This will make EmoconLite even more functional and versatile.BCI systems have a lot of potential as an alternative access method and as anaugmentative and alternative communication method for children with severe dis-abilities and/or medical complexities (Kinney-Lang et al., 2020; Brumberg et al.,2018). EmoconLite will support leisure experience for children with severe disabili-ties and/or medical complexities, giving them the opportunity to choose to initiatean activity and engage actively and independently through assistive technology. Assuch, continued development of BCIs for recreation, leisure and life pursuits will beessential in the well-being of these children. Through co-created participatory ac-tion research and action sensitive research, the interdisciplinary training programand the Therapeutic Recreation BCI Program at Holland Bloorview Kids Rehabil-itation Hospital will support further iterations of the development of EmoconLiteto meet client and family needs.Additionally, communication is a huge aspect of social experience as well as co-created leisure and recreation experience. Leisure interests, preferences, and choicecan be determined through connectivity and communication. These are essentialcomponents for participation, the development of leisure and life goals, and agency.Children acquiring BCI skills through leisure and play will become trained in
BCI, and these skills may be applicable to potential communication-based BCIsin the future. Development of the hardware and software, considerations of howBCIs can support recreation and leisure engagement, environmental controls, andcommunication are all areas that require co-created exploration to inform user-centred design. moconlite 9
This paper introduced a new free software application called EmoconLite, whichis an end-to-end solution that brings BCI applications to users with no program-ming experience. Building upon the Emotiv hardware and software system, userscan control external computer programs such as YouTube, HelpKidzLearn, BrainJoust, and Steam in real-time using their EEG signals. When integrated into Hol-land Bloorview’s BCI clinic, EmoconLite can support children with disabilitiesand/or medical complexities by enabling access to BCI technologies to help themadapt, benefiting their well-being and improving their quality of life. Strategies onmaintaining, motivating, mitigating frustration, and reducing device abandonmentwhile meeting clients’ needs to develop, learn and grow will need to be consideredalong with continued collaboration across the board.In the future, we will be extending the support for recognizing more mentaltasks, while expanding the repository of games and effector devices, connectingto Bluetooth toys and devices to provide extra functionalities to the users. Theseimprovements will be followed by making the next iteration of EmoconLite target-agnostic to control various applications. By expanding functionalities, a variety ofactivities will become available, opening up additional avenues for exploration.Consultation with and inclusion of clients, families and clinicians involved in theBCI programs will inform new iterations of EmoconLite to meet client interestsand needs. Through the BCI clinic, play and leisure is transformed. Children withsevere disabilities and their families will have an opportunity to adapt and copewith their congenital and acquired conditions through their brain signals, andexplore their potential, leading to resilience in the face of challenges while reducingbarriers to leisure participation.
Acknowledgements
This work was supported partially by Mitacs through the Mitacs El-evate program and Holland Bloorview Kids Rehabilitation Hospital Foundation. We want tothank Sarah House and Leslie Mumford at PRISM Lab, Heather Keating, Stephanie Hicks,Rachel Arsenault and Kendra Abbey at Holland Bloorview Kids Rehabilitation Hospital, Dr.Maureen Connolly at Brock University, and all members of the BCI-CAN network for theirconstructive feedback and suggestions.
Conflict of interest
The authors declare that they have no conflict of interest.
References
BCI Games. , 2019. Accessed: 2020-12-11.Brain Joust. , 2020. Accessed: 2020-12-11.
J. S. Brumberg, K. M. Pitt, A. Mantie-Kozlowski, and J. D. Burnison. Brain–computer interfaces for augmentative and alternative communication: A tuto-rial.
American journal of speech-language pathology , 27(1):1–12, 2018.Cortex API. https://emotiv.gitbook.io/cortex-api/ , 2019. Accessed: 2020-12-11.
EmotivBCI. , 2018. Accessed: 2020-12-11.HelpKidzLearn. , 2016. Accessed: 2020-12-11.C. D. Hood and C. P. Carruthers. Strengths-based tr program development usingthe leisure and well-being model: Translating theory into practice.
TherapeuticRecreation Journal , 50(1), 2016.M. Huber, J. A. Knottnerus, L. Green, H. van der Horst, A. R. Jadad,D. Kromhout, B. Leonard, K. Lorig, M. I. Loureiro, J. W. van der Meer, et al.How should we define health?
Bmj , 343, 2011.R. L. Jessup et al. Interdisciplinary versus multidisciplinary care teams: do weunderstand the difference?
Australian Health Review , 31(3):330, 2007.G. King, B. E. Gibson, B. Mistry, M. Pinto, F. Goh, G. Teachman, and L. Thomp-son. An integrated methods study of the experiences of youth with severe dis-abilities in leisure activity settings: the importance of belonging, fun, and controland choice.
Disability and Rehabilitation , 36(19):1626–1635, 2014.E. Kinney-Lang, D. Kelly, E. D. Floreani, Z. Jadavji, D. Rowley, E. T. Zewdie,J. R. Anaraki, H. Bahari, K. Beckers, K. Castelane, et al. Advancing braincomputer interface applications for severely disabled children through a multi-disciplinary national network: Summary of the inaugural pediatric bci canadameeting.
Frontiers in Human Neuroscience , 14:530, 2020.J. N. Mak and J. R. Wolpaw. Clinical applications of brain-computer interfaces:current state and future prospects.
IEEE reviews in biomedical engineering , 2:187–199, 2009.R. Mietola, S. Miettinen, and S. Vehmas. Voiceless subjects? research ethics andpersons with profound intellectual disabilities.
International Journal of SocialResearch Methodology , 20(3):263–274, 2017.L. Pritchard, S. Phelan, A. McKillop, and J. Andersen. Child, parent, and clinicianexperiences with a child-driven goal setting approach in paediatric rehabilita-tion.
Disability and Rehabilitation , pages 1–8, 2020.PySide2. https://pypi.org/project/PySide2/ , 2018. Accessed: 2020-12-11.Python 3.7.0. , 2018.Accessed: 2020-12-11.K. Ramella, B. Tennis, and C. Yoshioka. Making leisure choices using enablingswitches.
Annual in Therapeutic Recreation , 17:17–82, 2009.P. Rosenbaum and J. Gorter. The ‘f-words’ in childhood disability: I swear this ishow we should think!
Child: care, health and development , 38(4):457–463, 2012.A. Sayer.
Why things matter to people: Social science, values and ethical life . Cam-bridge University Press, 2011.C. A. Sexton. The overlooked potential for social factors to improve effectivenessof brain-computer interfaces.
Frontiers in systems neuroscience , 9:70, 2015.Steam. https://store.steampowered.com/ , 2003. Accessed: 2020-12-11.S. Steinert, C. Bublitz, R. Jox, and O. Friedrich. Doing things with thoughts:Brain-computer interfaces and disembodied agency.
Philosophy & Technology , 32(3):457–482, 2019.J. B. Wise. Integrating leisure, human flourishing, and the capabilities approach:
Implications for therapeutic recreation.
Therapeutic Recreation Journal , 52(3):254–268, 2018.J. Zhang, Z. Jadavji, E. Zewdie, and A. Kirton. Evaluating if children can usesimple brain computer interfaces.