Early Autism Spectrum Disorders Diagnosis Using Eye-Tracking Technology
Anna Solovyova, Sergiy Danylov, Shpenkov Oleksii, Aleksandr Kravchenko
AAbstract
While the number of children with diagnosed autism spectrum disor-der (ASD) continues to rise from year to year, there is still no universalapproach to autism diagnosis and treatment. A great variety of differenttools and approaches for the on-site diagnostic are available right now,however, a big percent of parents have no access to them and they tendto search for the available tools and correction programs on the Internet.Lack of money, absence of qualified specialists, and low level of trust to thecorrection methods are the main issues that affect the in-time diagnoses ofASD and which need to be solved to get the early treatment for the littlepatients. Understanding the importance of this issue our team decidedto investigate new methods of the online autism diagnoses and developthe algorithm that will be able to predict the chances of ASD accordingto the information from the gaze activity of the child. The results thatwe got during the experiments show supported our idea that eye-trackingtechnology is one of the most promising tools for the early detection ofthe eye-movement features that can be markers of the ASD. Moreover,we have conducted a series of experiments to ensure that our approachhas a reliable result on the cheap webcam systems. Thus, this approachcan be used as an additional first screening tool for the home monitoringof the early child development and ASD connected disorders monitoring.The further development of eye-tracking based autism diagnosis has a bigpotential of usage and can be further implemented in the daily practicefor practical specialists and parents. a r X i v : . [ c s . H C ] A ug arly Autism Spectrum Disorders DiagnosisUsing Eye-Tracking Technology Anna Solovyova, Sergiy Danylov, Shpenkov Oleksii, Aleksandr KravchenkoAugust 25, 2020
Autism Spectrum Disorder (ASD) is is a developmental disability characterizedby persistent impairments in social interaction as well as the presence of re-stricted, repetitive patterns of behavior, interests, and activities of the patient.The approach to diagnose autism has changed dramatically over the past 50years. At the beginning of 2000s, the common name autism has covered a widerrange of behavioral, communication and social disorders also referred to by theumbrella term ASD, which includes autistic disorder, Asperger’s syndrome andother related conditions [1].The number of patients with the different types of ASD symptoms continuesto grow and now 1 of every 68 children in Europe is diagnosed with autism. ASDaffects children from different countries and social statuses and it is reportedto occur in all racial, ethnic, and socioeconomic groups. There are a lot oftheories about the causes of such rocket grow of the children with diagnosedASD, however ahead of the other theories stands the fact that in the last decadethe number of the available test tools that allow to identified ASD has raiseddramatically, so the probability to diagnose ASD in children also has raised.But despite the fact of the widespread usage of test tools, a lot of childrenremain non-diagnosed and statistical data about the percentage of the childrenwith ASD remains biased due to the lack of available data from the developingcountries. The main problems that caused such disproportion in the autismdiagnosis are the absence of the practical specialists in the city, high cost of thetest programs, and different difficult conditions of the children which enable thepossibility to make the off-line testing. The inability to diagnose ASD at an earlyage is the other problem that leads to the negative consequences in the autismtreatment. It is commonly known that early diagnosis and early correctionprograms show good results in autism treatment and can help eliminate themost severe development of ASD [2]. On average parents observe the first sightsof autism at the age of 2 when the child establishes first social contacts withother children; however, in the majority of cases the ASD diagnoses age is morethan 4 years old. It results in the situation that on average each child loses two2rucial years in the ASD treatment that caused further difficulties in the child’sadaptation in the social life.It was showed that children with ASD have difficulties when needed to lookdirectly into the partner eyes, understand the facial expression and social in-teractions [3]. For instance, in the majority of cases, the patients with ASDhave trouble distinguishing fear from surprise, as well as to detect more diffi-cult emotions[4]. Human faces serve as an important part of the child’s socialadaptation and the correct perception of facial expressions is fundamental tothe development of social communication [5]. But the available tools for theASD diagnosis in most cases are not able to predict how good is the child inrecognizing different social situations based on understanding the core humanemotions. Today as a part of the ASD symptoms treatment applied behavioralanalyses are used, which can lead to good results in children when started at anearly age [6].Moreover, according to the data atypical eye movements of the individualswith ASD towards various stimuli can be a good source of the information aboutthe current state of the child development and can give good results in early ASDprediction [7], [8]. Early gaze patterns recognizing can has a big advantage forthe early ASD detection and further development of the individual correctionprogram. Moreover, the flexibility of this tool and absence of the need foradditional equipment can open access for the diagnostic tools for parents allaround the globe and thus increase the possibility for the success of correctionprograms among children of different ages.
The inability to correctly detect human emotions and intentions is one of themost crucial aspects of the autism spectrum disorder, that leads to numerousproblems in the social life of the patient. Though the individuals with ASDusually have different kinds of symptom severity and intellectual functioning,all of them have difficulties in everyday social interaction such as eye contact,engaging in reciprocal interactions, and responding to the emotional cues ofother people [9]. The early signs of autism such as failure to detect their ownname and no interest to the other people around can be recognized during thefirst year of life [10]. From the 2 to 3 additional social-oriented problems aredeveloped - a child usually has problems with social orienting, eye contact,joint attention, imitation, responses to the emotional displays of others, andface recognition [6], [11]. However, due to the big variety of accompanyingproblems, issues about the personal development path of each particular child,and limitations of the available diagnosis approaches, these problems remainedundetected up to 3 or 4 years. This delay in diagnosis and treatment of autismmakes the process of socialization way harder and significantly decreases thepossibility of the success correction programs.One of the most promising approaches of the autism diagnosis is the usageof eye-tracking technology to detect the deviations in the gaze behavior. Lack3f eye contact stands among the most typical symptoms of ASD; it usually ob-served among adults and children and leads to difficulties in the emotion recog-nition and socialization of the person. For example, during the experiment withemotional faces it was shown that adults with ASD primarily looking 2 timesless on the eye region of faces while focusing 2 times more on the mouth, body,and object regions relative to age- and verbal IQ matched controls [12]. Whilegoing through the structured viewing tasks, such as looking at still faces withsome kind of the particular emotions, the gaze fixation of the adults with ASDon atypical or non-feature areas of the face was also increasing ( for instance,gazing on cheeks, chin, or hairline but not at the eyes) [13]. In recent yearsa lot of the studies have been conducted to recognize the ASD children’s gazepatterns and their results show that the correlation between the gaze point andthe gaze fixation is also inherent to the children with diagnosed ASD. For in-stance, several studies have shown that children with ASD when seeing facespay more attention to the core features of the face such as eyes and nose, incomparison to typical individuals [14]. When observing the faces of the peoplewith pronounced emotions children with autism tend to pay maximal attentionto the eyes and mouth than other parts of the face [15]. In his recent workA. Klin states that altered gazing patterns such as less looking at the eyes andmore at mouth, body, and object areas can serve as an indicator of the reliablequantifiers of social disability and altered engagement with the social world inautism [16].It should be also mentioned that not all of the conducted to this momentstudies have shown the unequivocal support to the hypotheses of the correlationbetween gaze behavior and ASD. During the series of experiments McPartlandtogether with a colleague haven’t found the connection between gaze fixationbetween adults with ASD and typically developed adults [13]. Thus, this areaof research is yet a new branch in the ASD diagnoses that need further researchto understand to what extend the eye-tracking technology can be applied to theASD diagnosis.
The main challenge that we faced during the experiment preparation was thenecessity to make the comfort test zone for the children, which will help themconcentrate on the screen and will be not boring to completely lost their willing-ness to take part in the experiment. Thus, to make the experiment comfortablefor parents and children the screening process was held on the basis of privateclinics in a cozy silent room with the soft light. To monitor the experiment thepractical specialist together with a member of our team was present to ensurethat the screening process was conducted correctly. Children aged from 4-6were accompanied at all times by a parent or primary caregiver. To begin theexperimental session, a participant and caregiver entered the laboratory roomwhile a member of our team set up the application and eye-tracker. The childwas buckled into a comfortable seat mounted on a pneumatic lift so that view-4ng height (line of sight) was standardized due to the determined level of theparticipant’s eyes. The lights in the room were muted so that the child willbe able to concentrate only on the computer monitor. During the experimentsession, the experimenter (a member of our team) was concealed from the childsview but was able to monitor the conduction of the experiment as well as thegaze patterns of the child (Figure 1).Figure 1: During the setup for the experimentGaze fixation patterns were measured using eye-tracking equipment, de-scribed in the Data Acquisition and Analyse section. At the beginning of thedata collection process, the child was comfortably seated in the chair watchinga few special videos for the eye-tracker calibration. This video contains differ-ent shapes and figures of black and white color which appeared randomly onthe screen. While testing the children with ASD the average amount of theadditional calibration procedure was about 5 times (in the case of the high ac-tivity of the child during the examination session). To make the examinationmore relaxed for the children with ASD in most cases they were seated togetherwith the parent or a care-giver. When testing the children with typical devel-opment(TD) no additional calibration was used and they were seated alone onthe chair.During the screening session, the children observed different kinds of hu-man faces (both male and female) with different kinds of emotions (here weshould add the source of these images and why exactly they were used for ourexperiment). Besides the actor faces the photo of the family members of eachparticular child was also added to the video, so we were able to estimate thedifference between the perception of the unknown faces vs faces of the caregiver,whom the child sees every day. At the end of the experiment practical specialisttogether with members of our team made a short review for each of the parentsto help them better understand the results of their child and gain their feedbackabout the procedure. 5
Participants
Taking into account that the aim of our experiment was to gain the dataset of thegaze points of the children with ASD in compare to the gaze behavior patternsof the typically developed children, all the participants of the experiment weredivided into two groups: children with ASD (earlier diagnosed by the practicalspecialist) and typically developed children. The age range of the participantswas from 3 to 10 years old.To enlarge the local community of the ASD children and to make our datasetas wide as possible, we decided to post the invitation for the screening amongthe dedicated to the children’s ASD groups on the Facebook social platform.We have received about 150 requests for the experiment participation, but as wehave limitations of the eye-tracking due to the severity of the ASD symptoms,we were able to choose only 50 participants with ASD among them.Typically developed children were screened on the basis of the private school”Afina” according to the previous agreement with the management of the school.The experiment session was complying with all specified screening requirements(described in the previous section). Before the start of the experiment, all par-ents of the children that took part in the screening were aware of the procedureand the goals of the experiment. The age groups for the TD children were thesame as for the children with ASD to get the most comparable data.
Over the past 10 years, due to the appearance of high-quality cameras in laptopsand smartphones, eye-tracking has become possible on built-in cameras. How-ever, it has limitations in accuracy. In particular, for this experiment, the errorcould be up to 5per cent in both x and y coordinates, for a standard webcamthat shoots at 720p. Since the data was collected on a professional eye tracker,Gaussian noise was added to simulate the data from the built-in camera. Timedistributions by zones on the data with and without the noise you can see inFigure 2. Figure 2: Average time distribution per zones6s we can see the bars of the diagram for the noised data approached themean values, but the overall picture has not changed. Thus, it became inter-esting to check whether it is possible to train the algorithm on noisy data andwhat accuracy it will show. However, we cat not use fixations as input features,since we did not have them for noisy data. We could have merged the generateddata into fixations manually, but since the data is artificial, for the correctlyselected hyperparameters, we would get exactly the same fixations as for rawdata. To test the hypothesis, we used a simple fully connected network. Asinput data, we use time distribution by zones for each respondent. Since thedataset was very small and has only 5 features, it makes no sense to use thetraining results as a ready-made algorithm, but it was interesting to analyze thelearning process of the neural network. You can see the classification error andquality for clean and noisy data in Figure 3 (a, b)Figure 3: Model loss and model accuracy on clean and noised data per trainingepochsAccording to Figure 3, we see that the learning process for noised data isslower and there is no way to achieve 100per cent accuracy. However, it canbe seen that even on data with noise, we can classify children with sufficientlyhigh accuracy with only 5 features. Thus, it makes sense to conduct a similarexperiment on a conventional built-in camera and build an algorithm for morefeatures.During the experiment, we have observed marked differences in the gazebehavior of the children with ASD in comparison to the typically developedchildren. The core difference lay in the duration of the visual fixation on theface region of the observed actor (actress). It is known that that looking at aperson’s face is accompanied by a specific pattern of gaze movement [17]. Themost crucial point for face scanning and thus emotion recognition such as eyesand mouth of the person were our primary focus of observation. Any deviationfrom the standard gaze pattern can be a sign of the disability to understandand evaluate the context of the situation properly.As you can see in Fig.2 and Fig.3 children with diagnosed ASD tend to paymore attention to the objects outside of the face zone, while gaze points on themouth and eyes are very short.Children with typical development look more often on the face of the personrather than on the objects around the face and the time of the gaze fixationsshows the interest of the children on the mouth and eyes of the dialogue partner7igure 4: Example of the gaze fixation on the unknown face (ASD child)Figure 5: Example of the gaze fixation on the known face (ASD child))(Fig. 4 and Fig. 5). This fact indicates that typically developed children caneasily understand the emotions of the person and the contest of the situation,using the face of the person as the representation of the inner intentions. How-ever, children with ASD tend to miss up the face signals of the dialogue partner,which leads to the difficulties of emotion recognition and understanding.Figure 6: Example of the gaze fixation on the unknown face (TD child)
Understanding the gaze behavior of the children and measuring the deviationsin the individual gaze pattern of the child can be a good source of the infor-mation for the clinical diagnosis of ASD. The data that we have got during theexperiment sessions prove the theory that children with autism spectrum dis-orders tend to avoid direct gazing to the face of the dialogue partner and tendto spend more time observing the objects on the periphery. The most impor-tant source of the social and emotional information of the person such as eyesand mouth remain unobserved by the ASD children that lead to an incorrectunderstanding of the socio-emotional context of the particular situation and asa result of the difficulties in the day-to-day interactions with other people. Itwas also demonstrated that children with ASD have the similar problems whenlooking to the faces of the family members (in our experiment it was moms andnannies), so the communications within the family can be also difficult due tothe limited understanding of the emotions and feelings of the family members.8igure 7: Example of the gaze fixation on the known face (TD child))The main problem that we faced during the first stage of the experimentis the limitation of the available hardware. The eye-tracking systems as theones that were used in our experiment (add the description of the eye-tracker)are mainly used in the research laboratories, but they almost nit available forthe family home-using. In-build video cameras that are used in phones andnotebooks in most cases are not useful for tiny gaze movements detection and touse them the user needs to make eye-calibration a few times during the session.This limits the usage of the algorithm in the non-laboratory environment, butnew cameras that are used in the new generations of smartphones can be goodsupport for this ASD detection approach in the future.It should also be mentioned that eye-tracking methodology can be used asa kind of tool for the individual correction programs for children with autism.Thus, a lot of the application available today give a big variety of the emo-tion recognition task, they don’t track the gaze activity of the child during theworking sessions, so it gives almost no data about the developing of the gazebehavior of the child with ASD during the simulation sessions.However, to this date the proposed algorithm of the ASD diagnosis cant beused as a separate tool for the clinical results, it can be a good screening toolfor the early understanding of the possible autism spectrum disorders cases.Creating the basis for further clinical observations can be a good supportiveoption for the practical specialist. To achieve better more precise results inthe ASD prognoses our team plans to gather more experiment data during thenext stage of the research and use some modifications in the video screeningscenario that will allow us to use a wider range of emotions and simulate real-life situations.
References [1] Karen Weintraub. The prevalence puzzle: Autism counts.
Nature , 479:22–4, 11 2011.[2] Lynn Kern Koegel, Robert L. Koegel, Kristen Ashbaugh, and Jessica Brad-shaw. The importance of early identification and intervention for childrenwith or at risk for autism spectrum disorders.
International Journal ofSpeech-Language Pathology , 16(1):50–56, 2014.[3] Deborah Christensen, Matthew Maenner, Deborah Bilder, John Con-stantino, Julie Daniels, Maureen Durkin, Robert Fitzgerald, Margaret9urzius-Spencer, Sydney Pettygrove, Cordelia Rosenberg, JosephineShenouda, Tiffany White, Walter Zahorodny, Karen Pazol, and PatriciaDietz. Prevalence and characteristics of autism spectrum disorder amongchildren aged 4 years early autism and developmental disabilities monitor-ing network, seven sites, united states, 2010, 2012, and 2014.
MMWR.Surveillance Summaries , 68:1–19, 04 2019.[4] Madeline Harms, Alex Martin, and Gregory Wallace. Facial emotion recog-nition in autism spectrum disorders: A review of behavioral and neuroimag-ing studies.
Neuropsychology review , 20:290–322, 09 2010.[5] Yamamoto J. Matsuda S, Minagawa Y. Gaze behavior of children with asdtoward pictures of facial expressions.
Autism Res Treat. , 2015.[6] Geraldine Dawson, Sara Jane Webb, and James McPartland. Understand-ing the nature of face processing impairment in autism: Insights from be-havioral and electrophysiological studies.
Developmental neuropsychology ,27:403–24, 02 2005.[7] H. Duan, G. Zhai, X. Min, Y. Fang, Z. Che, X. Yang, C. Zhi, H. Yang,and N. Liu. Learning to predict where the children with asd look. In , pages704–708, 2018.[8] Tamami Nakano, Kyoko Tanaka, Yuuki Endo, Yui Yamane, Takahiro Ya-mamoto, Yoshiaki Nakano, Haruhisa Ohta, Nobumasa Kato, and ShigeruKitazawa. Atypical gaze patterns in children and adults with autism spec-trum disorders dissociated from developmental changes in gaze behaviour.
Proceedings. Biological sciences / The Royal Society , 277:2935–43, 10 2010.[9] Geraldine Dawson, Karen Toth, Robert Abbott, Julie Osterling, Jeff Mun-son, Annette M. Estes, and Jane Liaw. Early social attention impairmentsin autism: Social orienting, joint attention, and attention to distress.
De-velopmental psychology , 40:271–83, 04 2004.[10] Emily Werner, Geraldine Dawson, Julie Osterling, and Nuhad Dinno. Briefreport: Recognition of autism spectrum disorder before one year of age: Aretrospective study based on home videotapes.
Journal of Autism andDevelopmental Disorders , 30:157–162, 01 2000.[11] Peter Mundy, Marian Sigman, Judy Ungerer, and Tracy Sherman. Definingthe social deficits of autism: The contribution of non-verbal communicationmeasures.
Journal of Child Psychology and Psychiatry , 27:657 – 669, 091986.[12] Ami Klin, Warren Jones, Robert Schultz, Fred Volkmar, and Donald Co-hen. Defining and quantifying the social phenotype in autism.
The Amer-ican journal of psychiatry , 159:895–908, 07 2002.1013] Noah J Sasson Kevin A Pelphrey, Gregory Paul Jeffrey S Reznick, andJoseph Piven Barbara Davis Goldman. Visual scanning of faces in autism.
Journal of Autism and Developmental Disorders , 32:249–261, 2002.[14] Ami Klin, Warren Jones, Robert Schultz, Fred Volkmar, and Donald Co-hen. Visual fixation patterns during viewing of naturalistic social situationsas predictors of social competence in individuals with autism.
Archives ofgeneral psychiatry , 59:809–16, 10 2002.[15] J.N. Geest, C Kemner, M.N. Verbaten, and H Engeland. Gaze behaviorof children with pervasive developmental disorder toward human faces: Afixation time study.
Journal of child psychology and psychiatry, and allieddisciplines , 43:669–78, 08 2002.[16] Warren Jones, Katelin Carr, and Ami Klin. Absence of preferential lookingto the eyes of approaching adults predicts level of social disability in 2-year-old toddlers with autism spectrum disorder.
Archives of general psychiatry ,65:946–54, 08 2008.[17] Kano Fumihiro and Masaki Tomonaga. Face scanning in chimpanzees andhumans: continuity and discontinuity.