Extrapolating continuous color emotions through deep learning
Vishaal Ram, Laura P. Schaposnik, Nikos Konstantinou, Eliz Volkan, Marietta Papadatou-Pastou, Banu Manav, Domicele Jonauskaite, Christine Mohr
EExtrapolating continuous color emotions through deep learning
Vishaal Ram a , Laura P. Schaposnik (cid:63),b , Nikos Konstantinou c , Eliz Volkan d , MariettaPapadatou-Pastou e , Banu Manav f , Domicele Jonauskaite g , Christine Mohr g ( (cid:63) ) Corresponding author: [email protected] By means of an experimental dataset, we use deep learning to implement an RGB extrapolation ofemotions associated to color, and do a mathematical study of the results obtained through this neuralnetwork. In particular, we see that males (type m individuals) typically associate a given emotionwith darker colors while females (type f individuals) with brighter colors. A similar trend wasobserved with older people and associations to lighter colors. Moreover, through our classificationmatrix, we identify which colors have weak associations to emotions and which colors are typicallyconfused with other colors. Keywords: Color-associations, emotions, neural network
I. INTRODUCTION
The relation between colours and human emotion hasbeen studied for more than a century (e.g., see for in-stance [1–8]). Even longer ago, colours were commonlyassociated to emotions in a universal manner that al-lowed populations to understand quickly the given emo-tions. For example, for centuries in many cultures it hasbeen said that someone “had the blues” [29] or “is feelingblue” when being down or sad. As explained in [9], thephrase “feeling blue” comes from deepwater sailing ships:If a ship lost the captain or any of the officers during itsvoyage, then blue flags would be shown, and a blue bandwould be painted along the entire hull when returning tohome port.Inspired by [10, 11] we consider their data base [12]to analize the correlation between colours and emotionsvia a deep learning approach. Whilst machine learningtechniques have been used before in this direction (e.g.see [13] and references therein), we take a novel approachwhich allows us to discern several interesting patters.When using a deep learning approach to quantify color-emotion associations, one expects to observe certain be-haviors. In particular:(a) Certain colors should have strong associations withemotion and a high classification accuracy.(b) Some colors are associated with multiple emotionswould have a low classification accuracy.(c) Regional and geographic factors may have a factorin color emotion associations and would provide adeep learning approach to distinguish region.(d) Several colors are associated with a single emotion.Colour association studies usually consider a discretenumber of colours. In particular, this is the case of thestudy leading to the dataset [12] which we shall use inthe present paper, where participants associated emo-tions to 12 colour terms: red, orange, yellow, green, blue,turquoise, purple, pink, brown, black, grey, and white. Itshould be emphasised that the experiment did not showcolours but rather gave the terms of colours and it wasleft to the participants imagination the choice of whatthose words meant. To carry out our mathematical study, we have usedthe standard Decimal Code (R,G,B) to represent the 12colours of [12], a depiction of which is in Figure 1.
FIG. 1: A depiction of the 12 colors used in [12].
In the last decades colours have also been studied interms of emotional reactions to color hue, saturation, andbrightness (e.g., [14, 15]). Here, we shall put the twoapproaches together to consider a novel path, where welet the colour association within our neural network takea continuum of colours, hence considering a continuousRGB analysis [30], depicted in Figure 2.
FIG. 2: A depiction of the continuous RGB palate used inthe present paper.
After introducing some background in Section II, wededicate Section III and Section IV to the main findingsof our work, which can be seen in two different directionsin terms of associations of individuals of type m (males)and of type f (females): • From the classification matrix: we used a neuralnetwork approach which is different from the pa-per’s SVM approach [13]: – From the matrix we can identify which colorshave weak associations to emotions and whichcolors are typically confused with other colors. • Through an RGB regression we study single emo-tion associations as well as how associations variedacross age and gender ( m and f ), which we havenot seen previously discussed through mathemati-cal models. In particular, we see that: – Males typically associated the same emotionswith darker colors than females. – Older people do associations to lighter colors.We shall expand on the analysis and applications of theabove findings in Section V. a r X i v : . [ q - b i o . Q M ] A ug II. BACKGROUND: A MACHINE LEARNINGAPPROACH TO QUANTIFY THE SPECIFICITYOF COLOR-EMOTION
The current research was inspired by the previous work[13], and thus we shall dedicate this section to review-ing some of the main results which shall prove useful forour research and for comparison with our results. Theirresearch proposes that color-specificity of emotion asso-ciations and country-specificity of color-emotion associa-tions can be measured using a multivariate pattern clas-sification approach. When classifying the data, the au-thors used an optimized SVM (support vector machine)approach with a 10-fold cross-validation (CV) to evaluateaccuracy and considered several classifiers. • The first classifier predicted color on the basis of20 ratings of color-emotion associations. The clas-sifier achieved an accuracy of 38.7% when testedon 4 countries, and achieved an accuracy of 30.4% when the classifier was applied to a data set of30 countries. True positive rate was the highest forblack and red, followed by brown, pink, and grey.Thus they elicited very specific associations. • FIG. 3: Example of the platform within the survey used tocollect the data in [12].
III. PREDICTING THE ASSOCIATEDCOLOURS
As mentioned before, the data we shall use in thepresent manuscript consists of 944 submissions from [12].We shall consider the gender variables as indicating par-ticipants’ type m (male) and f (female), and shall con-sider the standard Decimal Code (R,G,B) conversion ofthe colours’ words used in [12] to carry out our mathe-matical study. Including age and gender, a total of 22input variables have been used to predict the associatedcolor. The learning structure used was a neural networkwith two hidden layers of size 10, and an output layerwith 12 nodes, one for each color as depicted in Figure 4. FIG. 4: Diagram of the neural network used.
The predicted color is the output node with the great-est activation. The neural network was trained usinga training set of size 2000 and testing using a cross-validations set of size approximately 10000. We usedthe standard back-propagation algorithm to optimize theedge weight parameters of the neural network to improvethe classification accuracy on the cross-validation set.Their results are summarized in Figure 5, with classi-fication accuracy at about 33 %.
FIG. 5: Confusion Matrix.
A. Accuracy depending on colours
From our study we see different levels of accuracy ob-tained in our predictions, allowing us to infer the levelof emotional association that colours have. Indeed, thefollowing is observed, as depicted in Figure 6: • The colours black, brown, pink, and red are pre-dicted with high accuracy and indicate a strong as-sociation with emotion; • In contrast, the colors green and purple are pre-dicted with very low accuracy and indicate weakassociation with emotion.
FIG. 6: Emotional association within discrete colour analysis
B. Colour exchanges
Through our study we note that some pairs of colorsare frequently confused, indicating the variability of theemotional association, and the need for further under-standing the relation between colours in such pairs bothfrom the visual as well as emotional point of view. Inparticular we note that, as depicted in Figure 7, the fol-lowing occurs: • black and brown are confused for grey; • blue for turquoise and green; • orange for yellow; • brown for green and grey; • and white for turquoise. FIG. 7: Confusion exchanges
Interestingly, colors like green and purple are rarelypredicted at all, even though all colors appear the samenumber of times. This suggests that their association toemotion is weak enough to make it beneficial for the neu-ral network to always output a different color. In order toimprove accuracy, and given the above usual exchangesof colours, we consider the following two alternative setups to understand the colour association:(A) Group similar colors to minimize confusion and im-prove accuracy;(B) Implement a continuous RGB output instead of aclassification network. This can be used to demon-strate how subtle changes in color associations areinfluenced by varying emotion, and it should benoted that it was not attempted in [13].
C. Grouping Colors
To improve the consistency of the color classification,one natural step is to reduce the dimension of the out-put layer by grouping colors by similar emotion asso-ciations in accordance to Figure 8. In particular, wehave grouped together the darker colors black, grey, andpurple. By reducing the dimension by a factor of two,we have significantly improved the classification accu-racy to 52% from the 33% in the full color classifica-tion. One notable observation is that the brown groupis more often classified as the black-grey-purple groupand the blue-green-turquoise group than its own group, apattern not observed in the classification without dimen-sion reduction. Additionally, the white group is almostnever classified correctly, most often being classified asthe blue-green-turquoise group. One possible explana-tion being that white is mostly associated with relief,a trait that is shared among the blue-green-turquoisegroup. Lastly, the orange-yellow group was incorrectlyclassified as the blue-green-turquoise group about just asoften as it was correctly classified, which again can beexplained by shared emotion such as joy and pleasure.
FIG. 8: Confusion Matrix for Color Groupings.FIG. 9: Average Color-Emotion Association
D. Average colour association
In the survey, participants are asked to describe thestrength of the association between color-emotion pairson a scale from 0 to 5. The table shown in Figure 9displays the average association value across every color-emotion pair among all the participants in the data set,which serves as a visualization of the entire data set with-out the age and gender factors. Some of the limitationsof the results described in the single color association isthat certain emotions can strongly associate with multi-ple conflicting colors, leading to an ambiguous mix of col-ors, often producing a brown. For example, both amuse-ment and pleasure are associated with both colors likeblue and turquoise as well as colors like red and orange,the combination of which cannot be described by a singlecolor, leading to some ambiguity and variation.Certain colors such as black, pink, and red displaystrong associations with at least one emotion, whileothers, such as brown and yellow, display weakerassociations across all emotions. Likewise, we see thatcertain emotions such as love, relief, and sadness displaystrong associations with at least one color, while others,such as contempt, regret, and shame, display weakerassociations across all colors.
IV. CONTINUOUS RGB ANALYSIS
The use of a continuum of colours for studying humanemotional associations has appeared in the literature formany decades now (see, for example see the use withinneural networks in [16], within emotional expressions inrobotics [17], and when considering color combinations in[18]), and has become more important recently because oftheir impact in image retrieval processes. Whilst manyexperiments have used color emotion metrics for singlecolors or pairs of colors, in many cases similar metricswere recently used in image retrieval showing that hu-mans perceive color emotions for multi-colored images insimilar ways (e.g. see [19]). Hence, it becomes relevant toanalize the continuum of colours in the context of [12] todeduce novel correlations between colors and emotions.The RGB regression study was implemented with thesame neural network depicted in Figure 4, with 22 nodes,two hidden layers of size 20 and 10, and 3 output nodes,representing the intensity of red, green, and blue. Thetraining set (size 2000) was modified so that each colorwas converted to an RGB value based on standard con-ventions for color. By means of the neural network de-scribed above, we develop an interface which allows oneto predict the colour depending on the choice of differentvariables. This interface, shown in Figure 10, has a sliderfor each input variable which can be adjusted. The ageslider varies from 0 to 50, the gender slider varies from0 to 1 (0 being female while 1 male), and each emotionvaries from 0 to 5. The color at the top represents thepredicted color based on the RGB output.
FIG. 10: RGB Regression Interface
A. Color-emotion associations across gender
Gender differences when encoding and decoding colorassociations and facial emotions have long been consid-ered (e.g. see [20] and references therein). Whilst inmany cases differences have been found, in other settingsgender and age have not presented much difference. Forexample, gender and age were shown to be determiningfactors in the selection of achromatic black [21], whilsthere were no sex differences in the main emotions linkedto red within the children considered in [22].By adjusting the gender input in the interface of Figure10, one can compare color-emotion associations acrossgender as shown in Figure 11. Indeed, through the re-gression, we find that females associate admiration witha dark purple, while males a dark blue. Across mostemotions, we find: • both genders typically associate with similar emo-tions and colors. • exceptions include admiration, in which females as-sociate with a light purple while males with a blue, • and regret, which females associate with a darkpink while males with a red-orange.Through our study we also found that females typi-cally associate emotions with lighter shades. This is mostclearly seen in: FIG. 11: Gender (type m and f ) mathematical comparisonin single color associations (for Age = 20) • anger, compassion, disgust, guilt, pleasure, pride,relief, and sadness; • exceptions, in which cases males choose lighter col-ors, include shame and disappointment.Interestingly, this agrees with experiments done foryoung ages, where young boys were more likely to as-sociate positive emotions with darker colours than girls,e.g. see [22]. B. Color-emotion associations across age
By adjusting the age input, we can compare color-emotion associations across age as shown in Figure 12.Through our mathematical model, we find that: • younger people associate anger with a darker shadeof red, while older people a lighter shade; • older people associate most emotions with lightershades than younger people. Examples are anger,compassion, hate, interest, and love, • on the other hand, older people associate relief witha darker blue. • There exist some emotions for which we see a con-sistent color across the age spectrum such as con-tempt, guilt, joy, sadness, and shame.Some emotions shifts are associated with a more browncolor as age increases, as in admiration and contentment,which could indicate ambiguity as in multiple color as-sociations, or simply a lack of data as the age increasesleading to extrapolation. We shall return to this in Sec-tion V, where we shall compare our results to those fromexperimental papers which have looked into aging andcolor-emotional associations [23–25].
FIG. 12: Age comparison in single color associations (GenderNeutral), where shades of colours are obtained through thestandard Decimal Code (R,G,B) obtained through our algo-rithm associated to each emotion.
It is interesting to note that in the present manuscriptwe have considered responses from Greek speakers fromGreece, Greek speakers from Cyprus, Turkish speakersfrom Cyprus, and Turkish speakers from Turkey, andthus generically we get anger associated with shades ofred. However, this should not hastily be generalized toa broader population: indeed, there have been studiessuch as [26] showing that people from Poland connectedanger, envy, and jealousy with purple instead of red. Intheir paper it is suggested that “cross-modal associationsoriginate in universal human experiences and in culture-specific variables, such as language, mythology, and lit-erature” , and it would be indeed interesting to designa neural network to discern those differences once datafrom different regions is readily available.
C. Emotions associated to single colors.
Through our study, we can classify emotions into twodistinguished sets: those that have a distinct associationto a specific colour, and those that do not. Indeed, emo-tions such as anger, disgust, fear, love, pride, relief, andsadness have very clear and distinct associations to a spe-cific color. However, other emotions such as amusement,compassion, joy, and shame may be associated with mul-tiple color and therefore provide an ambiguous color, gen-erally a shade of brown or some arbitrary mix of colors.Figure 13 displays the color associated with a single emo-tion at maximum intensity among the gender neutral andaverage age setting. (e.g. pure relief is most associatedwith a light blue color while pure anger is associated witha red color).
FIG. 13: Single emotion associations predicted throughour model (Age = 20), where shades of colours are ob-tained through the standard Decimal Code (R,G,B) obtainedthrough our algorithm associated to each emotion.
V. CONCLUDING REMARKS
In the present paper we have analyzed a dataset pro-vided by the online survey [12] in which 944 participantswere presented with a series of 12 colors and asked todetermine the association of each color with a set of 20emotions in a scale from 0 to 5, via a platform depicted inFigure 3. We used a machine learning method to quan-tify the strength of color-emotions associations as wellas their variation across age and gender. To quantifythe strength of color-emotion associations, we employeda neural network to classify colors based the participants20 emotion association values as well as their age andgender. The network consists 22 input nodes and 12 out-put nodes (for each color), and two hidden layers of 10nodes, as depicted in Figure 4. The results of the clas-sification based on a cross-validation of 10000, and ona training set of 2000, are summarized by the confusionmatrix in Figure 5.In particular, we found that black, brown, pink, andred are classified correctly with high accuracy, indicatinga strong association with specific emotions, while greenand purple are rarely predicted and classified accurately,indicating a weak association with emotion (see Figure6). In order to improve accuracy, we combined simi-lar colors and formed 6 color groups to reduce confusionamong similar colors and ran an identical neural networkto classify each group based on their corresponding emo-tion associations, increasing the classification accuracy to52% (see Figure 8). In particular, we find that:(I) brown is typically confused with the other darkercolors, and;(II) both white and the orange-yellow group are notablyconfused with the blue-green-turquoise group.We expect this to be likely due to shared associationswith positive/negative emotions. We also investigateda neural-network based regression to associate emotionand age/gender parameters colors on a continuous RGBspectrum. We used a neural network with the same 22input nodes, but with 3 output nodes representing thered, green, and blue values. The network also consistedof 2 hidden layers of 20 and 10 nodes, and was trainedon 2000 examples from the survey. We implemented aninterface to display the output of the regression based onthe 22 input variables, as shown in Figure 10.Using the regression, we determined precise colors inDecimal Code (R,G,B) associated with single emotionsand its variation due to age and gender. The colors asso-ciated with single emotions (with age set to 20 and gen-der neutral) are summarized in Figure 13. Most colorswhich were associated with multiple (conflicting) colorslike amusement and shame produced ambiguous colors,generally a shade of brown or an arbitrary mix of colors.In Figure 11, we displayed the effects of gender on singleemotion associations. Through the regression, we showthat:(III) in most cases females associated the same emo-tion with lighter colors, with exceptions includingshame and disappointment.In particular, Figure 12 displaces the effects of age onsingle emotion associations. The different perception ofcolours and emotional associations have long been con-sidered (e.g., see [21, 22, 27]). In particular, from experi-mental data it is seen that bright colors have mainly pos-itive emotional associations, and dark colors have mainlynegative emotional associations when not taking genderinto account, but women responded more positively thanmen to bright colors, and they also respond more nega-tively to dark colors [27]. From our mathematical studyas seen in item (III) above, we see that females indeedchoose brighter colours than men for most positive emo-tions, and choose darker colors than men for the mostnegative emotions, which is in agreement with the exper-imental results. However, there are still some negativeemotions for which women tend to choose lighter coloursthan men, and thus it would be very interesting to peruse an experimental study of a detailed colour association toa large range of negative emotions across gender.When considering age differences, we found that olderpeople associate most emotions with lighter shades of thesame color, with relief being an exception. It is inter-esting to highlight the different effects of age on colourassociation:(IV) Some emotions such as contempt, guilt, joy, sad-ness, and shame were associated with similarshades across age.(V) On the other hand, some emotions like admirationand pleasure shift to a more brown color as ageincreases, indicating ambiguity or a lack of data(extrapolation).In the last decades, many studies have found that olderpeople are less able to identify facial expressions thanyoung people [23, 24]. Moreover, since it has been hy-pothesised that the emotional association of colours couldbe related to their association with facial expressions [25],one can expect the association of colour to get tintedwith age, leading to further ambiguity, as we have foundin our study both through items (I) and (V) above. Itwould be indeed very interesting to carry out experimen-tal research on how ageing leads to a higher probabilityof brown being associated with emotions.Finally, considering the findings in terms of cross-ageand gender colour association, one may consider their im-plications within marketing strategies of different kinds.An example of this is, for instance, when marketing per-fumes for different ages and genders. In such cases, onecan use the colours people tend to associate with theirsmells for the packaging [28], and we propose that furtherconsidering their association with emotions could lead tomore targeted audiences.When considering single color association, certain emo-tions strongly associate with multiple conflicting colors,leading to an ambiguous mix colors (often a brown) out-putted by the regression. This ambiguity can also ex-plain some of the variation found in the age and genderspectrum analysis. For example, both shame and admi-ration have relatively weak associations across all colors,which could explain some of the variation shown in thegender comparison. It should also be noted that in theRGB regression, different instances of the training leadto slightly different results in the colors produced. To re-duce variation, we train using a high number of iterations(1000) and increase the standard regularization parame-ter λ (incorporated to avoid the risk of risk of overfitting)to achieve the most consistent results. Acknowledgments.
The leading authors (*) VR andLPS are thankful to MIT PRIMES-USA for the oppor-tunity to conduct this research together, and in partic-ular Tanya Khovanova for her continued support andJames Unwin for insightful comments on a draft of themanuscript. Moreover, they greatly acknowledge thework of their co-authors to produce [12].
Contributions of authors.
Vishaal Ram and LauraSchaposnik carried out the research and preparation ofthe present manuscript. The remaining authors wereresponsible for the data source. In particular, NikosKonstantinou was responsible for Greek Cypriot dataof [12], Eliz Volkan was responsible for Turkish Cypriotdata of [12], Marietta Papadatou-Pastou was responsiblefor Greek data of [12], Banu Manav was responsible forTurkish data of [12] of [12], and Christine Mohr andDomicele Jonauskaite were esponsible for the overalldata conceptualisation of [12], coordination of the trans-lations, data acquisition, and research dissemination.
Funding.
The work of Laura Schaposnik is partiallysupported through the NSF grants CAREER DMS1749013. The Swiss National Science Foundationsupported the work of Domicele Jonauskaite with the Doc.CH fellowship grant (P0LAP1 175055) and Chris-tine Mohr with a project grant (100014 182138)
Affiliations. (a) Milton High School, Milton, GA 30004, USA.(b) University of Illinois, Chicago, IL 60607, USA.(c) Department of Rehabilitation Sciences Faculty ofHealth Sciences Cyprus University of Technology.(d) Cyprus International University, Nicosia, Cyprus.(e) National and Kapodistrian University of Athens,Athens, Greece.(f) Kadir Has University, Faculty of Art and Design,Department of Interior Architecture and EnvironmentalDesign.(g) Institute of Psychology, University of Lausanne,Switzerland. [1] M. Luckiesh,
The language of color (Dodd, Mead, 1918).[2] S. Mastandrea,
Howemotions shapeaesthetic experiences (Cambridge University Press, 2014), pp. 500–518, Cam-bridge Handbooks in Psychology.[3] G. H. Taylor, The Journal of Abnormal Psychology ,347 (1912).[4] L. B. Wexner, Journal of applied psychology , 432(1954).[5] F. M. Adams and C. E. Osgood, Journal of cross-culturalpsychology , 135 (1973).[6] K. NAz and H. Epps, College Student J , 396 (2004).[7] S. E. Palmer and K. B. Schloss, Proceedings of the Na-tional Academy of Sciences , 8877 (2010).[8] K. R. Scherer, V. Shuman, J. Fontaine, and C. Sori-ano Salinas (2013).[9] Origins of navy terminology , 101350(2019).[11] D. N. e. a. Jonauskaite D, Abu-Akel A, Psychol Sci. (inpress) (2020).[12] N. D. Domicele Jonauskaite, Christine Mohr (PI), fors-base (2019).[13] M. C. e. a. Jonauskaite D, Wicker J, Royal Society (2019).[14] P. Valdez and A. Mehrabian, Journal of experimentalpsychology: General , 394 (1994).[15] L. Wilms and D. Oberfeld, Psychological research ,896 (2018).[16] M.-F. Lee and G.-S. Chen, in (IEEE, 2013), pp. 433–438.[17] M.-g. Kim, H. S. Lee, J. W. Park, S. H. Jo, and M. J.Chung, in RO-MAN 2008-The 17th IEEE InternationalSymposium on Robot and Human Interactive Communi-cation (IEEE, 2008), pp. 219–224.[18] G. Csurka, S. Skaff, L. Marchesotti, and C. Saunders, in
Proceedings of the Seventh Indian Conference on Com- puter Vision, Graphics and Image Processing (2010), pp.298–305.[19] M. Solli and R. Lenz, Color Research & Application ,210 (2011).[20] N. G. Rotter and G. S. Rotter, Journal of NonverbalBehavior , 139 (1988).[21] B. Manav, Color Res Appl , 144 (2007).[22] C. J. Boyatzis and R. Varghese, The Journal of geneticpsychology , 77 (1994).[23] L. H. Phillips, R. D. MacLean, and R. Allen, The Jour-nals of Gerontology Series B: Psychological Sciences andSocial Sciences , P526 (2002).[24] N. C. Ebner and M. K. Johnson, Emotion , 329 (2009).[25] M. R. Zentner, Developmental Science , 389 (2001).[26] R. B. Hupka, Z. Zaleski, J. Otto, L. Reidl, and N. V.Tarabrina, Journal of cross-cultural psychology , 156(1997).[27] M. Hemphill, The Journal of genetic psychology , 275(1996).[28] H. N. Schifferstein and I. Tanudjaja, Perception33