Sentifiers: Interpreting Vague Intent Modifiers in Visual Analysis using Word Co-occurrence and Sentiment Analysis
SSentifiers: Interpreting Vague Intent Modifiers in Visual Analysis usingWord Co-occurrence and Sentiment Analysis
Vidya Setlur * Tableau Software
Arathi Kumar † Tableau Software. (a) (b)
Figure 1: Screenshots of
Sentifiers showing the interpretation of vague intent modifiers using sentiment analysis and word co-occurrence. Interactive text is displayed with the ability to adjust the ranges using slider widgets. (a) For a dataset of earthquakesin the US [38], the system associates the vague modifier “unsafe” with the data attribute magnitude . Similar negative sentimentpolarities (shown in red ) result in a top N filter of magnitude and higher to be applied. (b) A dataset showing the health and wealthof nations [20]. Here, the modifier “struggling” has a negative sentiment, while the incomePerCapita and lifeExpectancy attributeshave positive sentiments shown in blue . The diverging sentiment polarities result in Bottom N filters applied. A BSTRACT
Natural language interaction with data visualization tools often in-volves the use of vague subjective modifiers in utterances such as“show me the sectors that are performing ” and “where is a good neighborhood to buy a house?.” Interpreting these modifiers is oftendifficult for these tools because their meanings lack clear semanticsand are in part defined by context and personal user preferences. Thispaper presents a system called
Sentifiers that makes a first step inbetter understanding these vague predicates. The algorithm employsword co-occurrence and sentiment analysis to determine which dataattributes and filters ranges to associate with the vague predicates.The provenance results from the algorithm are exposed to the useras interactive text that can be repaired and refined. We conduct aqualitative evaluation of the
Sentifiers that indicates the usefulness ofthe interface as well as opportunities for better supporting subjectiveutterances in visual analysis tasks through natural language.
Keywords: vague and subjective modifiers, natural language inter-action, sentiment analysis, visual analysis.
Index Terms:
K.6.1 [Management of Computing and InformationSystems]: Project and People Management—Life Cycle; K.7.m[The Computing Profession]: Miscellaneous—Ethics * e-mail: [email protected] † e-mail: [email protected] NTRODUCTION
Understanding user intent in a query has been recognized as animportant aspect of any natural language (NL) interaction system [9,25]. Search queries typically consist of keywords and terms called modifiers that imply a diverse set of search intents [23]. Whilebasic keyword matches from users’ search queries might elicit areasonable set of results, interpreting modifiers provides a betterunderstanding of the semantics in the queries [26].Recently, NL interfaces for visual analysis tools have garneredinterest in supporting expressive ways for users to interact withtheir data and see results expressed as visualizations [1–3, 15, 19, 22,30, 35, 36]. Users often employ vague language while formulatingnatural language queries when exploring data such as “which countryhas a high number of gold medals?” or “what time of the day do more bird strikes occur?” [21]. There has been some precedenceof research to better understand how these simple vague modifierscomprising of superlatives and numerical graded adjectives shouldbe appropriately interpreted [21, 31]. However, users also employless concrete and often subjective modifiers such as ‘best’, ‘safe’,and ‘worse’ in utterances [21]. The interpretation of such modifiersmakes it challenging for natural language interfaces to preciselydetermine the extensions of such concepts and mapping intent to theanalytical functions provided in the visual analysis systems.
Contribution
This paper introduces
Sentifiers , a system to explore reasonableinterpretations and defaults for such subjective vague modifiers innatural language interfaces for visual analysis. The algorithm iden-tifies numerical attributes that can be associated with a modifierusing word co-occurrence. Sentiment analysis determines the filter The name
Sentifiers is a portmanteau of ‘sentiment’ and ‘modifier,’blending their concepts as they co-occur together. a r X i v : . [ c s . H C ] S e p anges applied to the attributes. Similar polarities result in associat-ing the Top N of data values for an attribute with the modifier, whilediverging polarities are mapped to the
Bottom N .Figure 1a indicates that ‘unsafe’ and the attribute magnitude have similar negative sentiment polarities, defaulting to a higherearthquake magnitude range as seen in the map. The system hasthe ability to utilize any domain-specific information if available,such as WolframAlpha [4]. Figure 1b shows diverging polarities forthe modifier ‘struggling’ paired with attributes incomePerCapita and lifeExpectancy . Lower numerical filter ranges based on thestatistical properties of the data are applied to generate the scatterplot.Interactive text is displayed to show the provenance of the system’sinterpretation with clickable portions exposed as widgets that canbe refined by the user. An evaluation of the system provides usefulinsights for future system design of NL input systems for supportingvague concepts in visual analysis.
ELATED W ORK
Research exploring the semantics of vague concepts for understand-ing intent transcends three main categories: (1) Computational Lin-guistics, (2) Intent and Modifiers in Search Systems, and (3) NaturalLanguage Interaction for Visual Analysis.
The notion of vagueness in language has been studied in the com-putational linguistics community [32]. Research has focused on theconceptualization and representation of vague knowledge [8]. The
Vague system introduces a technique for generating referring expres-sions that included gradable adjectives [39]. De Melo et al. inferadjective grade ordering from large corpora [14] and Vegnaduzzoautomatically detects subjective adjectives [40]. Computationallinguists have developed approaches for subjectivity and polarity la-beling of word senses [6, 42]. In our work, we draw inspiration fromlinguistic literature, specifically polarity identification for computingthe semantics around vague subjective concepts.
Search systems have explored techniques to deduce intent in queriesduring exploratory search. Several techniques exist to extract entity-oriented search intent to improve query suggestions and recommen-dations [16]. Detecting intent in search systems is also based onquery topic classification [33]. Bendersky et al. assign weights toterms in a search query based on concept importance [7]. Recentwork has focused on deriving query intent by fitting queries intotemplates [5, 25]. Li et al. employ semantic and syntactic featuresto decompose queries into keywords and intent modifiers [25]. Re-searchers have predicted search intent and intentional task types fromsearch behavior [12, 28]. While the goal of our work to interpret in-tent in queries is similar to that of search tasks, we focus on resolvingvague modifiers to generate relevant visualization responses.
Similar to search systems, natural language interfaces for visual anal-ysis need to understand intent and handle modifiers in the utterances.DataTone provides ambiguity widgets to allow a user to update thesystem’s default interpretation [19]. Eviza and Analyza supportsimple pragmatics in analytical interaction through contextual infer-encing [15, 30]. Evizeon [22] and Orko [35] extend pragmatics inanalytical conversation. None of these systems consider how impre-cise modifiers can be interpreted. The
Ask Data system describes thehandling of numerical vague concepts such as ‘cheap’ and ‘high’ byinferring a range based on the underlying statistical properties of thedata [31]. Hearst et al. explore appropriate visualization responsesto singular and plural superlatives and numerical graded adjectivesbased on the shape of the data distributions [21]. We extend thiswork to more vague, subjective modifiers. HE Sentifiers S YSTEM
We introduce a system,
Sentifiers that interprets vague modifierssuch as ‘safe’ and ‘struggling’ in a NL interface for visual analysis.The system employs a web-based architecture with the input queryprocessed by an ANTLR parser with a context-free grammar, similarto parsers described in [22,30]. A data manager provides informationabout the data attributes and executes queries to retrieve data. Thequery upon execution, generates a D3 visualization result [10].
The process for resolving a set of data attributes and their values toa modifier found in the NL input to
Sentifiers , is outlined as:
Algorithm 1:
Interpretation of Vague Modifiers in
Sentifiers
Input:
Natural language utterance α Output:
Generate visualization response α is the NL input utterance. m is the vague modifier in the utterance α .Part-of-Speech tagger POS identifies m in α . attrs num is the set of numerical attributes in the dataset D . attrs cnum is the set of co-occurring numerical attributes in D with attrs cnum ∈ attrs num . PMI computes co-occurrence scores w c for m and attrs num . polarity computes sentiment polarities p for m and attrs cnum . Invoke
POS ( α ) returning m . Compute
PMI ( m , attrs num ) → w c for each attr i ∈ attrs cnum . Compute polarity ( m , attrs cnum ) → p . Update interface based on w c and p . Vague modifiers are gradable adjectives that modify nouns and andare associated with an abstract scale ordered by their semantic inten-sity [24]. Gradable adjectives can be classified into two categoriesbased on their interpretation as measure functions [24]. Numericalgraded adjectives such as ‘large’ and ‘cheap’ are viewed as measure-ments that are associated with a numerical quantity for size and costrespectively. Complex graded adjectives like ‘good’ and ‘healthy’tend to be underspecified for the exact feature being measured.While the interpretation of numerical gradable adjectives hasbeen explored in NL interfaces for visual analysis [21, 30, 31], thispaper specifically focuses on the handling of complex gradableadjectives.
Sentifiers first applies a commonly used performant part-of-speech (POS) tagger during the parsing process to identify thesecomplex gradable adjectives and their referring attributes in theNL utterances [37]. The system can distinguish complex gradableadjectives by checking for the absence of superlative or comparativetags that are used to annotate numerical graded adjectives.
The next step maps the vague modifier to a scale based on its se-mantic intensity so that the modifier can be interpreted as a set ofnumerical filters for generating a visualization response. We baseour approach on linguistic models that represent the subjectivity ofcomplex modifiers as a generalized measure mapping the modifier tonumerical attributes in a multidimensional space [18]. For example,the subjectivity of the modifier ‘healthy’ can be interpreted based on‘weight’, ‘amount of exercise’, and ‘hospital visits.’
Sentifiers computes the semantic relatedness between the modifierand the numerical data attributes using a co-occurrence measure.To have sufficient coverage for co-occurrence, we use an extensiveGoogle n-grams corpus [27]. To maximize the chances of co-occurrence, Sentifiers considers co-occurrence between all n-gramcombinations of the modifier and the attribute names. For example, An n-gram is a contiguous sequence of n items from a sequence of text. igure 2: PMI values for the modifier ‘struggling’ with each of theattribute n-grams, ‘income’, ‘life expectancy’, and ‘population’ in theGoogle n-gram corpus. Higher PMI scores indicate a higher co-occurrence of the modifier and attribute terms. some of the n-grams for the attribute income per capita are‘income per capita,’ ‘income per,’ ‘per capita,’ and ‘income.’We employ a Pointwise Mutual Information Measure (PMI), aninformation-theoretic measure that quantifies the probability of howtightly occurring a modifier m and a numerical attribute attr num areto the probability of observing the terms independently [13]. Wefound this measure to work well and was performant with terse wordco-occurrence pairings without requiring sentence embeddings. Weconsider any numerical attribute attr cnum that has a non-zero PMIscore, indicating the presence of a co-occurrence with m . The PMIof modifier n-gram t m with one of the attribute n-grams t attr is: PMI ( t m , t attr ) = log p ( t m , t attr ) p ( t m ) p ( t attr ) (1) Once the modifier is semantically associated with co-occurring nu-merical attributes, we need to determine a reasonable numericalrange to associate with the modifier. Sentiment polarity analysisis a linguistic technique that uses positive and negative lexicons todetermine the polarity of a phrase [43]. The technique provides theability to dynamically compute the sentiment of the phrase based onthe context in which its terms co-occur rather than pre-tagging thephrase with absolute polarities, which is often not scalable.
Figure 3: Sentiment polarity logic with sentiments and their normalizedscores for the modifiers and the numerical attributes. (a) The modifier‘safe’ and attribute earthquake magnitude have positive and negativesentiments respectively, resulting in a
Bottom N range based on theRichter scale [4]. (b) The modifier ‘booming’ and attribute income percapita both have positive sentiments, resulting in a
Top N computedbased on statistical data properties.
We determine the individual sentiment scores with a sentimentclassification based on a recursive neural tensor network [34]. Wechoose this technique as its models handle negations and reasonablypredict sentiments of terser phrases, characteristic of queries to
Sentifiers . The sentiments are returned as a 5-class classification:very negative, negative, neutral, positive, and very positive. Thevalues are normalized as [ − , + ] , ranging from negative to positiveto provide an overall sentiment. We then determine the sentimentpolarities of the modifier m and co-occurring attribute attr cnum pairbased on their individual sentiments (ignoring the strength of thesentiments) using the following combinatorial logic. We treat neutralsentiment similar to positive sentiment as neutral text tends to lie nearthe positive boundary of a positive-negative binary classifier [43]. if ( sentiment m == positive or sentiment m == neutral ) and ( sentiment attr cnum == positive or sentiment attr cnum == neutral ) then Compute
TopN ( attr cnum ) . else if ( sentiment m == positive or sentiment m == neutral ) and sentiment attr cnum == negative then Compute
BottomN ( attr cnum ) . else if sentiment m == negative and ( sentiment attr cnum == positive or sentiment attr cnum == neutral ) then Compute
BottomN ( attr cnum ) . else if sentiment m == negative and sentiment attr cnum == negative then Compute
TopN ( attr cnum ) . end if Sentifiers uses sentiment polarities to compute the ranges in twoways: If domain knowledge exists, the system uses the information todetermine a default (Figure 3a uses the Richter scale [4]). Otherwise,the system computes
Top N to range from [ med + MAD , max ] and Bottom N to range from [ min , abs ( med − MAD )] where med , MAD , min , and max are the median, median absolute deviation, minimum,and maximum values for attr cnum respectively (see Figure 3b). Wechoose MAD as it tends to be less affected by non-normality [11].
Figure 1 shows the
Sentifiers interface with an input field that acceptstext queries. Upon execution of the query, range filters for the co-occurring numerical attributes are applied, showing a visualizationresponse. The system interpretation is expressed in the form ofinteractive text [41] above the visualization (Figure 4a) to help theuser understand the provenance of how the modifier was interpreted.Positive, negative, and neutral sentiments are shown in blue , red ,and yellow respectively (Figure 4b). The text contains widgets thatshow ranges starting from the highest co-occurring one. Similar toother NL systems [19, 30, 31], we expose system presumptions aswidgets (Figure 4c). If domain-specific semantics are used, a linkto the source is provided (Figure 1a). To provide easier readability, Sentifiers displays up to two widgets. Word co-occurrence andsentiment analysis techniques can result in incorrect results. Theuser has the ability repair the system decisions (Figures 4d and f)and the interface updates to reflect the changes (Figure 4e). Theserefinements are persistent for the duration of the user session.
VALUATION
We conducted a user study of
Sentifiers with the following goals:(1) collect qualitative feedback on the handling of the modifiers forvarious visual analysis tasks and (2) identify system limitations. Thisinformation would help provide insights as to how the handling ofcomplex vague modifiers could integrate into a more comprehensiveNL visual analysis interface. The study was exploratory in naturewhere we observed the types of vague modifiers people asked andhow they responded to the system behavior. Because the main goalof our study was to gain qualitative insight in the system behavior,we encouraged participants to think aloud with the experimenter. igure 4: Interactive text response to a query “which countries are booming ?”.
Sentifiers provides the ability to refine the system defaults.
We recruited ten volunteers (five males, five females, age 24 – 65).All were fluent in English and all regularly used some type of NLinterface such as Google. Eight used a visualization tool on a regularbasis and the rest considered themselves beginners.
Each participant was randomly assigned a dataset of earthquakesin the US [38] or the health and wealth of nations [20] with equalnumber of participants for each. We began with a short introductionof how to use the system. Participants were instructed to phrasetheir queries in whatever way that felt most natural and to tell uswhenever the system did something unexpected. We discussedreactions to system behavior throughout the session and concludedwith an interview. The study trials were done remotely over a sharedvideoconference to conform with social distancing protocol due toCOVID-19. All sessions took approximately 30 minutes.
We employed a mixed-methods approach involving qualitative andquantitative analysis, but considered the quantitative analysis mainlyas a complement to our qualitative findings. The quantitative analysisconsisted of the number of times participants used vague subjectivemodifiers and interacted with the text response.
Overall, participants were positive about the system and identifiedmany benefits. Several participants were impressed with the abilityof the system to understand their queries (“I typed scary to see whatit would do, and it understood.” [ P Sentifiers ’ text feedback wasfound to be helpful (“I wasn’t sure how the system would handlethis, but it was pretty clear when I saw the response” [ P P µ = .
7) with a total of 24 unique complex modifiersoverall. The three most common modifiers were ‘good’, ‘bad’,‘severe’ for the earthquakes dataset and ‘prosperous’, ‘flourishing’,‘poor‘ for the health and wealth of nations dataset. All participantsinteracted with the text response to understand the system behavior.The most common interaction was updating the data ranges forthe attributes (69% of the interactions), followed by adding newattributes (23%), and deleting attributes from the interpreted result(8%). Comments relevant to this behavior included, “The rangeseemed high for me and I changed it. It was nice to see the systemremember that” [ P P P Support for more complex interpretations:
The current imple-mentation does not support combinations of vague modifiers in thesame query. For example, the system was unable to interpret “showme countries that are doing very well and poorly .” [ P P Sentifiers failed to cor-rectly interpret queries such as “which countries are reasonablydoing well ,” where P Handling customization and in-situ curation:
The topic of cus-tomization of the interpretation behavior came up during the study.For example, P af-fordable and it showed me an income range. I was expecting aresponse that considered inflation, GDP or have a way for me todefine that.” The algorithms employed in Sentifiers assume that thedata attributes are curated with human-readable words and phrases.However, data is often messy with domain-specific terminology.Future work should explore mechanisms for users to customize se-mantics of attributes and interpretations in the flow of their analysis.
Handling system expectations, biases, and failures:
NL algo-rithms have shown to exhibit socio-economic biases, including gen-der and racial assumptions often due to the nature of the trainingdata [17]. Their use can perpetuate and even amplify cultural stereo-types in NL systems. For example, P good places to live and the system responded with a high incomeper capita. To me, that opens up bigger issues such as gentrificationand economic segregation.” This suggests that there is a responsi-bility for improved transparency in system behavior; determiningappropriate de-biasing methods remains an open research problem. ONCLUSION
This paper presents a technique to explore how a system can interpretsubjective modifiers prevalent in natural language queries duringvisual analysis. Using word co-occurrence and sentiment polarities,we implement
Sentifiers to map these modifiers to more concretefunctions. We expose the provenance of the system’s behavior asan interactive text response. An evaluation of the system indicatesthat participants found the system to be intuitive and appreciated theability to refine the system choices. Feedback from interacting with
Sentifiers identifies opportunities for handling vagueness in languagein the future design of such natural language tools to support dataexploration. As Bertrand Russell stated [29] – “Everything is vagueto a degree you do not realize till you have tried to make it precise.”
EFERENCES [1] IBM Watson Analytics. .[2] Microsoft Q & A. https://powerbi.microsoft.com/en-us/documentation/powerbi-service-q-and-a/ .[3] ThoughtSpot. .[4] WolframAlpha: Computational Intelligence. .[5] G. Agarwal, G. Kabra, and K. C.-C. Chang. Towards rich queryinterpretation: Walking back and forth for mining query templates. In
Proceedings of the 19th International Conference on WWW , WWW’10, pp. 1–10. ACM, 2010.[6] C. Akkaya, J. Wiebe, A. Conrad, and R. Mihalcea. Improving theimpact of subjectivity word sense disambiguation on contextual opinionanalysis. In
Proceedings of the Fifteenth Conference on ComputationalNatural Language Learning , CoNLL ’11, pp. 87–96. Association forComputational Linguistics, USA, 2011.[7] M. Bendersky, D. Metzler, and W. B. Croft. Learning concept im-portance using a weighted dependence model. In
Proceedings of theThird ACM International Conference on Web Search and Data Mining ,WSDM ’10, pp. 31–40. ACM, 2010.[8] F. Bobillo and U. Straccia. Fuzzy ontology representation using OWL2.
International Journal of Approximate Reasoning , 52(7):1073–1094,2011. Selected Papers - Uncertain Reasoning Track - FLAIRS 2009.[9] J. Bos. Computational semantics in discourse: Underspecification,resolution, and inference.
Journal of Logic, Language, and Information ,13(2):139–157, 2004.[10] M. Bostock, V. Ogievetsky, and J. Heer. D3: Data-driven documents.In
IEEE Transactions on Visualization and Computer Graphics , 2011.[11] P. Cairns.
Doing Better Statistics in Human-Computer Interaction .Cambridge University Press, 2019. doi: 10.1017/9781108685139[12] Z. Cheng, B. Gao, and T.-Y. Liu. Actively predicting diverse searchintent from user browsing behaviors. In
Proceedings of the 19th Inter-national Conference on WWW , WWW ’10, pp. 221–230. Associationfor Computing Machinery, New York, NY, USA, 2010.[13] K. W. Church and P. Hanks. Word association norms, mutual infor-mation, and lexicography. In , pp. 76–83. ACL, Vancouver, BritishColumbia, Canada, June 1989. doi: 10.3115/981623.981633[14] G. de Melo and M. Bansal. Good, great, excellent: Global inference ofsemantic intensities.
Transactions of the Association for ComputationalLinguistics , 1:279–290, 2013. doi: 10.1162/tacl a 00227[15] K. Dhamdhere, K. S. McCurley, R. Nahmias, M. Sundararajan, andQ. Yan. Analyza: Exploring data with conversation. In
Proceedings ofthe 22nd International Conference on Intelligent User Interfaces , IUI2017, pp. 493–504, 2017.[16] H. Duan and C. Zhai. Mining coordinated intent representation forentity search and recommendation. In
Proceedings of the 24th ACMCIKM , CIKM ’15, pp. 333–342. Association for Computing Machinery,New York, NY, USA, 2015. doi: 10.1145/2806416.2806557[17] B. Friedman and H. Nissenbaum. Bias in computer systems.
ACMTrans. Inf. Syst. , 14(3):330–347, July 1996.[18] W. Galit and G. W. Sassoon. Multidimensionality in the grammar ofgradability, 03 2016.[19] T. Gao, M. Dontcheva, E. Adar, Z. Liu, and K. G. Karahalios. Data-Tone: Managing ambiguity in natural language interfaces for datavisualization. In
Proceedings of the 28th Annual ACM Symposium onUser Interface Software Technology , UIST 2015, pp. 489–500. ACM,New York, NY, USA, 2015.[20] Gapminder. Health and wealth of nations. , 2020.[21] M. Hearst, M. Tory, and V. Setlur. Toward interface defaults for vaguemodifiers in natural language interfaces for visual analysis. In , pp. 21–25, 2019.[22] E. Hoque, V. Setlur, M. Tory, and I. Dykeman. Applying pragmaticsprinciples for interaction with visual analytics.
IEEE Transactions onVisualization and Computer Graphics , 24(1):309–318, 2017.[23] B. J. Jansen, D. L. Booth, and A. Spink. Determining the user intent ofweb search engine queries. In
Proceedings of the 16th International Conference on WWW , WWW ’07, pp. 1149–1150. Association forComputing Machinery, New York, NY, USA, 2007. doi: 10.1145/1242572.1242739[24] C. Kennedy.
Projecting the Adjective: The Syntax and Semantics ofGradability and Comparison . Outstanding dissertations in linguistics.Garland, 1999.[25] X. Li. Understanding the semantic structure of noun phrase queries. In
ACL . Association for Computational Linguistics, July 2010.[26] C. D. Manning, P. Raghavan, and H. Sch¨utze.
Introduction to Informa-tion Retrieval . Cambridge University Press, USA, 2008.[27] J.-B. Michel, Y. K. Shen, A. P. Aiden, A. Veres, M. K. Gray, , J. P.Pickett, D. Hoiberg, D. Clancy, P. Norvig, J. Orwant, S. Pinker, M. A.Nowak, and E. L. Aiden. Quantitative analysis of culture using millionsof digitized books.
Science , 331(6014):176–182, 2011. doi: 10.1126/science.1199644[28] M. Mitsui, C. Shah, and N. J. Belkin. Extracting information seekingintentions for web search sessions. In
Proceedings of the 39th Inter-national ACM SIGIR Conference on Research and Development inInformation Retrieval , SIGIR ’16, pp. 841–844. Association for Com-puting Machinery, New York, NY, USA, 2016. doi: 10.1145/2911451.2914746[29] B. Russell.
The Analysis of Mind . Library of philosophy. G. Allen &Unwin Limited, 1921.[30] V. Setlur, S. E. Battersby, M. Tory, R. Gossweiler, and A. X. Chang.Eviza: A natural language interface for visual analysis. In
Proceedingsof the 29th Annual Symposium on UIST , pp. 365–377. ACM, 2016.[31] V. Setlur, M. Tory, and A. Djalali. Inferencing underspecified naturallanguage utterances in visual analysis. In
Proceedings of the 24thInternational Conference on Intelligent User Interfaces , IUI ’19, pp.40–51. ACM, New York, NY, USA, 2019.[32] S. Shapiro.
Vagueness in Context . Oxford University Press, 2006.[33] D. Shen, J.-T. Sun, Q. Yang, and Z. Chen. Building bridges for webquery classification. In
Proceedings of the 29th Annual InternationalACM SIGIR Conference on Research and Development in InformationRetrieval , SIGIR ’06, pp. 131–138. Association for Computing Ma-chinery, New York, NY, USA, 2006. doi: 10.1145/1148170.1148196[34] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng,and C. Potts. Recursive deep models for semantic compositionalityover a sentiment treebank. In
Proceedings of the 2013 Conference onEmpirical Methods in Natural Language Processing , pp. 1631–1642.Association for Computational Linguistics, Oct. 2013.[35] A. Srinivasan and J. Stasko. Orko: Facilitating multimodal interactionfor visual exploration and analysis of networks.
IEEE transactions onvisualization and computer graphics , 24(1):511–521, 2018.[36] Y. Sun, J. Leigh, A. Johnson, and S. Lee. Articulate: A semi-automatedmodel for translating natural language queries into meaningful visual-izations. In
International Symposium on Smart Graphics , pp. 184–195.Springer, 2010.[37] K. Toutanova, D. Klein, C. D. Manning, and Y. Singer. Feature-rich part-of-speech tagging with a cyclic dependency network. In
Proceedings of the 2003 ACL Conference , NAACL ’03, pp. 173–180.Association for Computational Linguistics, USA, 2003.[38] USGS. Earthquake facts and statistics. , 2020.[39] K. van Deemter. Generating vague descriptions. In
Proceedings ofthe First International Conference on Natural Language Generation- Volume 14 , INLG ’00, pp. 179–185. Association for ComputationalLinguistics, USA, 2000. doi: 10.3115/1118253.1118278[40] S. Vegnaduzzo. Acquisition of subjective adjectives with limited re-sources. In
In Proceedings of the AAAI Spring Symposium on ExploringAttitude and Affect in Text: Theories and Applications , 2004.[41] B. Victor. Explorable explanations. http://worrydream.com/ExplorableExplanations/ , 2011.[42] J. Wiebe and E. Riloff. Creating subjective and objective sentenceclassifiers from unannotated texts. In
Proceedings of the 6th CiCLing ,CICLing ’05, pp. 486–497. Springer-Verlag, Berlin, 2005.[43] T. Wilson, J. Wiebe, and P. Hoffmann. Articles: Recognizing contextualpolarity: An exploration of features for phrase-level sentiment analysis.