A Large-Scale, Automated Study of Language Surrounding Artificial Intelligence
AA Large-Scale, Automated Study of Language Surrounding Artificial Intelligence
Autumn Toney
Center for Security and Emerging TechnologyGeorgetown [email protected]
Abstract
This work presents a large-scale analysis of artifi-cial intelligence (AI) and machine learning (ML)references within news articles and scientific pub-lications between 2011 and 2019. We imple-ment word association measurements that automat-ically identify shifts in language co-occurring withAI/ML and quantify the strength of these word as-sociations. Our results highlight the evolution ofperceptions and definitions around AI/ML and de-tect emerging application areas, models, and sys-tems (e.g., blockchain and cybersecurity ). Recentsmall-scale, manual studies have explored AI/MLdiscourse within the general public, the policy-maker community, and researcher community, butare limited in their scalability and longevity. Ourmethods provide new views into public perceptionsand subject-area expert discussions of AI/ML andgreatly exceed the explanative power of prior work.
You shall know a word by the company it keeps.J. R. Firth
Referenced in a wide-range of domains, from science fic-tion to autonomous vehicles, artificial intelligence (AI) hasgained significant attention from societies and governmentsworldwide. Despite its emerging prominence in the pub-lic sphere, AI still lacks a consistent, universally-accepteddefinition, making it a challenging subject to analyze overtime [Bryson, 2019; Cave et al. , 2018; Chuan et al. , 2019;Fast and Horvitz, 2017; Krafft et al. , 2020; Legg and Hutter,2007]. Recent studies conducted surveys and manual annota-tion tasks to understand how subject-area experts, the generalpopulation, and policy makers define and perceive AI [Fastand Horvitz, 2017; Krafft et al. , 2020; Cave et al. , 2018; Cave et al. , 2019; Chuan et al. , 2019; Russell and Norvig, 2002;Sweeney, 2003]. While these studies are a necessary prelim-inary step in uncovering historical and current perceptions ofAI, these studies are limited in scalability and constrained tothe period of time in which they were performed.To improve on these limitations and gain new insights, we present a large-scale, automated approach to analyze the lan-guage surrounding AI in the public sphere during any timeperiod. Using text corpora from news articles and scientificpublication abstracts, we analyze AI references in two do-mains: one that represents public perceptions of AI and onethat represents subject-area expert applications of AI. Ouranalysis includes more than 170,000 AI-related news articlesand 77,000 AI-related scientific publication abstracts. To thebest of our knowledge, our work is the largest-scale study onAI references in the public sphere, specifically in text corpora.Our approach uses word association structures in text cor-pora and measures the strength of and the shifts in word as-sociations over time. In psycholinguistics, word associationstructures are often identified through human studies whereparticipants are presented with a word (e.g., coffee ) and re-spond with a word that comes to mind (e.g., mug ). Word as-sociation structures in text corpora can be automatically iden-tified by analyzing words that frequently co-occur within adesignated proximity of each other. Thus, word associationstructures can be automatically derived from a text corpus,indicating the different characteristics of a word by the “com-pany it keeps.”We use mutual information to measure the strength of as-sociation and a normalized co-occurrence frequency valueto measure the shifts in frequently co-occurring words overtime. Mutual information is an indicator of words that havea high probability of exclusively co-occurring with a targetword. In this way, mutual information values identify wordsthat have an exclusive co-occurrence with a target word, asopposed to words with a general co-occurrence. For exam-ple, in news articles, we find that robotics has a high co-occurrence frequency with artificial intelligence and machinelearning and a high mutual information value, whereas big has a high co-occurrence frequency but a low mutual infor-mation value. Shifts in co-occurrence frequency ranks indi-cate words that are emerging, decreasing in frequency, or in-creasing in frequency. For example, in scientific publicationabstracts, we find that convolutional is an emerging word co-occurring with artificial intelligence and machine learning .In the following sections, we provide a background onword association structures (Section 2), summarize relatedwork (Section 3), describe the datasets studied in our analysis(Section 4), define our methodology (Section 5), and presentand discuss our experimental results (Sections 6 and 7). a r X i v : . [ c s . C L ] F e b Background
In psychology, the law of mental association defines the phe-nomena of learning by contiguity, a learning process that as-sociates a stimulus and response based on their frequency andproximity (e.g., coffee being associated with mug) [James,1890]. Applied to linguistic theory, the law of mental asso-ciation relates to language acquisition; words associated toa particular concept are stored closely in a human’s “mentallexicon” [Dobel et al. , 2010]. When words frequently co-occur, by some definition of proximity, their association ina mental lexicon is strengthened [Wettler and Rapp, 1993;Church and Hanks, 1990].Word associations are dynamic, as language evolves as-sociations will change [Nelson et al. , 2004]. Prior psy-cholinguistic studies identify word association norms acrosspopulations [Nelson et al. , 2004; Wettler and Rapp, 1993;Church and Hanks, 1990; Buchanan et al. , 2019]. These stud-ies commonly use priming—showing a stimulus (an imageor word)—and measure the speed of a response or the con-sistency of responses across the participants. For example,Nelson et al. conducted a free response survey where partic-ipants were asked to write the first word that came to mindafter reading a cue word [Nelson et al. , 2004]. This surveywas designed to capture associative knowledge and charac-teristics of meaning; responses were shown to be affectedby culture and trends. Consistent word associations acrossparticipants indicate a common experience with words, andinconsistent word associations across participants highlightexperiences that vary from the norm [Nelson et al. , 2004].These human surveys are translated to automated proce-dures performed on text corpora, providing a scalable analy-sis of word associations, by defining word co-occurrences astwo words appearing within a designated window size of eachother [G¨unther et al. , 2016]. Window size defines a proxim-ity constraint for word co-occurrence; for example, a windowsize of two considers only two words to the left and two wordsto the right of the target word. Wettler and Rapp find that awindow size of five is optimal for large text corpora, as itdoes not dilute the language surrounding a target word andmaintains a close enough proximity to capture true associa-tion [Wettler and Rapp, 1993].Word co-occurrences, measured by using a specified win-dow size, have been used in natural language processingtasks, such as generating semantic spaces [Lund and Burgess,1996]. In practice, applying word association methods onlarge-scale text corpora eliminates the sample bias of partic-ipants, as participant judgements are used to measure norms.However, word association methods do not eliminate othertypes of biases captured in linguistic norms, though theyhave also proven useful in this space [Caliskan et al. , 2017;Bolukbasi et al. , 2016].
Previous studies have taken various manual approaches to de-fine AI and present public perceptions of AI. Russell andNorvig analyzed AI defintions in eight textbooks publishedbetween 1978 and 1993, and then specified four main waysAI is defined: 1) think like humans, 2) act like humans, 3) think rationally, and 4) act rationally [Russell and Norvig,2002]. Building on Russell and Norvig’s work, Sweeneymanually categorized 996 AI-related publications cited byRussell and Norvig [Sweeney, 2003]. Sweeney found that987 of these publications favor defining AI in terms of ratio-nal thinking and rational behavior [Sweeney, 2003].Cave et al. surveyed 1,078 UK participants and collectedresponses from multiple choice and free response questionsto learn about public perceptions of AI [Cave et al. , 2019].Notably, 85% of respondents stated that they had heard ofAI before, with 25% of them defining AI in terms of robots.Krafft et al. conducted two surveys, one with 98 partici-pants and one with 86 participants, where the authors askedAI researchers what they consider AI systems to be and howthey define AI in practice [Krafft et al. , 2020]. They com-pared the survey responses to policy definitions of AI, whichthey collected by manually annotating 83 policy documentsfrom 2017 through 2019 [Krafft et al. , 2020]. Krafft et al.found that policy documents typically use “human-like” def-initions of AI, wheres AI researchers define AI through tech-nical problems and functionality [Krafft et al. , 2020].Fast and Horvitz analyzed AI-related news articles fromthe New York Times between 1986 and 2016, approximately3 million articles in total [Fast and Horvitz, 2017]. Any para-graph in an article that mentioned the terms artificial intelli-gence , AI , or robot was selected, reducing the data down to8,000 paragraphs over the thirty years. The paragraphs weremanually annotated by Amazon Mechanical Turkers, and theresults describe trends in the public perception of AI overtime. Specifically, mentions of AI have increased, the gen-eral population has become more optimisitc about AI, andconcerns over the loss of control of AI are increasing [Fastand Horvitz, 2017]. Chuan et al. sampled news articles fromLexisNexis and ProQuest from five U.S. news sources (USAToday, The New York Times, Los Angeles Times, New YorkPost, and Washington Post) that contain the term artificial in-telligence . Using stratified sampling, they reduced the 2,485AI-related articles to 399 articles that are manually annotatedby three graduate students. Chuan et al.’s study focused moreon understanding the framing of AI in news articles and pre-sented findings on the main topics, cited sources, and senti-ment in their subset of AI-related news articles. They foundthat AI was mainly discussed in Business and Economy and
Science and Technology article topics and that AI ethics isincreasingly discussed [Chuan et al. , 2019].
We study two large-scale datasets to generate subsets ofAI/ML text data: 1)
AI/ML N
EWS , 170,858 news arti-cles from the LexisNexis database [LexisNexis, 2020] and 2)
AI/ML A
BSTRACTS , 77,880 scientific publication abstractsfrom the Microsoft Academic Graph [Sinha et al. , 2015]. Wecategorize an article or abstract as AI/ML if it contains theterms artificial intelligence or machine learning at least once,using Bryson’s description of important terms for understand-ing AI [Bryson, 2019]. For both news articles and scien-tific publication abstracts, we normalize the text by settingall words to lower case and removing symbols, digits, URLs,mail addresses, phone numbers, and punctuation except forapostrophes. Additionally, we remove all stop words usingNLTK’s English set of stop words. LexisNexis Database:
The LexisNexis database containsnews article texts that were published between 2011 and2020. We analyze English-language articles from 2011, 2015,and 2019 that were published by sources of good editorialquality. LexisNexis generates source editorial rankings fornews articles on a rank scale is from 1 to 5, with 1 beinghigh quality (e.g., The New York Times) and 5 being lowquality (e.g., message boards). We select news articles thathave an editorial rank of 1, 2, or 3, which includes interna-tional, national, business, regional, industry, and governmentnews sources. We use the duplicate ID assigned by Lexis-Nexis to de-duplicate the articles. The 170,858 news arti-cles in AI/ML N
EWS is comprised of these filtered and de-duplicated documents.Table 1 provides details for each year’s subset ofAI/ML N
EWS . Over time, the number of documents, to-kens (unique vocabulary words), and sources significantly in-crease. Figure 1 displays the counts of artificial intelligence and machine learning mentions in AI/ML N
EWS . Men-tions of artificial intelligence are more frequent than mentionsof machine learning over the entire period of study; thereare 2,446 AI mentions and 554 ML mentions in 2011 and187,066 AI mentions and 103,175 ML mentions in 2019.
Year Num. ofDocuments Avg. WordCount Num. ofTokens Num. ofSources
EWS corpus N u m be r o f M en t i on s Artificial IntelligenceMachine Learning
Figure 1: Term counts for artificial intelligence and machine learn-ing respectively in AI/ML N
EWS over time
Microsoft Academic Graph:
Microsoft Academic Graph(MAG) contains scientific research publication documentsfrom eight categories: Book, Book Chapter, Conference,Dataset, Journal, Patent, Repository, and Thesis [Sinha et al. ,2015]. We use a subset of MAG documents from 2011, 2015,and 2019 that contain an abstract in their publication record.Table 2 provides details for each year’s subset ofAI/ML A
BSTRACTS . There are comparatively fewer wordsper text instance and fewer documents in AI/ML A B - STRACTS than in AI/ML N
EWS . Figure 2 displays the countsof artificial intelligence and machine learning mentions in AI/ML A
BSTRACTS . Mentions of machine learning aremore frequent than mentions of artificial intelligence over theentire period of study, the opposite of AI/ML mentions inAI/ML N
EWS . In 2011, there are 6,210 ML mentions and3,012 AI mentions, and in 2019, there are 59,006 ML men-tions and 22,414 AI mentions.
Year Num. ofDocuments Avg. WordCount Num. ofTokens
BSTRACTS corpus N u m be r o f M en t i on s Artificial IntelligenceMachine Learning
Figure 2: Term counts for artificial intelligence and machine learn-ing respectively in AI/ML A
BSTRACTS over time
We use two word association measurements to provide a com-prehensive understanding of how words co-occurring with ar-tificial intelligence and machine learning change over time:mutual information and normalized co-occurrence rank. Bothmeasurements rely on a definition of co-occurrence, thus wedefine co-occurrence as a word co-occurring within a windowsize of the terms artificial intelligence and machine learning .Since AI and ML are two-word terms, we consider wordsto the left of artificial/machine and words to the right of in-telligence/learning within the defined window size. We ac-count for edge cases in selecting co-occurring words, suchas artificial intelligence or machine learning ending a doc-ument. Figure 3 demonstrates term co-occurrences within afive-word window under various text positions. We define mutual information between two words accordingto Church and Hanks [Church and Hanks, 1990]. Given twowords, w and w , their mutual information I ( w , w ) is de-fined as: I ( w , w ) = log Pr[ W = w , W = w ]Pr[ W = w ] Pr[ W = w ] (1) Pr[ W = w ] is the probability that a word drawn at randomfrom a document in the text corpus is equal to w . Specifically, Pr[ W = w ] = w count total count , where w count is the number oftimes that w appears in the document and total count is thenumber of words in the document. Pr[ W = w , W = w ] is the joint probability that the two words co-occur (withina window size) in a text corpus, indicating association. Iftwo words frequently co-occur in text, Pr[ W = w , W = . The term has at least five words preceding it and five words following it:... w l w l w l w l w l artificial intelligence w r w r w r w r w r ...2. The term has less than five words preceding it, following it, or both: w l w l artificial intelligence w r w r w r w r w r ...... w l w l w l w l w l artificial intelligence w r w r . w l w l artificial intelligence w r , w r .3. The term either starts or ends the article:Artificial intelligence w r w r w r w r w r ...... w l w l w l w l w l artificial intelligence. Figure 3: Examples of co-occurring words within a designated win-dow size of five to artificial intelligence (AI). The yellow highlightindicates words considered to be co-occurring with AI. w ] will be a larger value than if they infrequently co-occur;thus, a stronger association between two words (frequent co-occurrences) results in a larger value for I ( w , w ) . In thisway, mutual information quantifies the strength of association between word co-occurrences and provides a metric that canbe compared over time. In order to identify words that emerge or are increasing ordecreasing in their frequency of co-occurrence over time, wedefine a normalized co-occurrence rank value. Normaliza-tion is necessary for frequency rank comparisons over time,since there is a significant increase of tokens from year to yearin both AI/ML N
EWS and AI/ML A
BSTRACTS (see Table1 and 2). We identify the set of co-occurring words withinthe designated window size and count their frequency of co-occurrence with either term ( artificial intelligence or machinelearning ) in a given year. We sort the co-occurring frequencyword set in descending order of frequency. This ordering as-signs a rank value to each word, with the most frequently co-occurring word at rank . We compute the normalized rankby dividing the assigned ranks by the total number of wordsin the co-occurring frequency word set in a given year. For aword w and a year y , where rank w is the assigned rank of theword w and total y is the number of words in the year’s co-occurring frequency word set, the normalized rank N ( w, y ) is defined as: N ( w, y ) = rank w total y (2)We compute the normalized ranks of co-occurring wordsfor each year (2011, 2015, and 2019), rounding the normal-ized ranks to the nearest 0.05 to smooth the results. Then wecompute the standard deviation ( σ ) of normalized ranks foreach word over the three years. Words with lower σ valuesshift minimally from year to year, indicating words that main-tain their co-occurrence frequency over time. Words withhigher σ values shift maximally from year to year, indicat-ing words that emerge or have a downward or upward co- occurrence frequency trend over time. In this way, normal-ized co-occurrence rank identifies the shift of language overtime. Our first step in experimentation is to compute the word co-occurrence frequencies, using a window size of five as rec-ommended by Wettler and Rapp [Wettler and Rapp, 1993],for each year respectively. We also tested windows with sizesthree and eight in our experiments, but found that the resultsvary minimally, thus we present the results for window sizeof five (see Supplementary Materials for details). The com-puted word co-occurrence frequencies provides us with thenecessary data to apply our word association measurements.For each year’s results, we sort the words by descend-ing order of their frequency. Table 3a and 3b display thetop 15 most frequently co-occurring words with AI/ML overtime in AI/ML N
EWS and AI/ML A
BSTRACTS respectively.We find that AI/ML A
BSTRACTS have a more consistentset of top 15 co-occurring words over time, with minimalwords being introduced or being dropped in each year, com-pared to the top 15 co-occurring words in AI/ML N
EWS . InAI/ML N
EWS , four out of the 15 words consistently appear,such as technology and data . In AI/ML A
BSTRACTS , 10out of the 15 words consistently appear, such as techniques and algorithms . We find that data and using appear in bothAI/ML N
EWS and AI/ML A
BSTRACTS for all three years.
We measure the strength of association for words co-occurring with artificial intelligence and machine learning using mutual information (described in Section 3.2). Wecompute mutual information for words that have a relativefrequency of at least 0.1% for each year respectively to limitour analysis to popular words. Table 4 presents mutual infor-mation (MI) and relative frequency (Frq) for the top five co-occurring words with the highest mutual information valueover time in AI/ML N
EWS and AI/ML A
BSTRACTS . AI/ML N
EWS : We find that ai , algorithms , and robotics consistently appear in the top five words with highest MI val-ues, indicating that these words have a consistent and strongassociation to the terms artificial intelligence and machinelearning in news articles. Interestingly, siri and azure appearin Table 3a, highlighting that these systems are disproportion-ately represented in the context of AI/ML in the news articlecorpus. Other words with high mutual information to artifi-cial intelligence and machine learning change from year toyear, with words like mit and stanford dropping in mutual in-formation from the 2011 results to the 2019 results and wordslike blockchain and cybersecurity appearing first in the 2019results. AI/ML A
BSTRACTS : We find that ai , classifiers , and su-pervised consistently appear in the top five words with high-est mutual information values, indicating that these wordshave a strong association to the terms artificial intelligence and machine learning in AI/ML A BSTRACTS . The words uci and repository reference the UCI Machine LearningRepository, a data repository which currently warehouses 559
011 • computer, technology , ai , science, software, research, data , techniques, using , uses, use, algorithms, robotics,said2015 • data , technology , ai , analytics, big, new, using , computer,technologies, science, research, said, algorithms, robotics,also, human2019 • ai , data , technology , technologies, intelligence, analytics,artificial, using , new, use, big, digital, learning, company,internet (a) AI/ML N EWS techniques , data , methods , based , using , algorithms , used , method , paper, learning , approach, system ,research, mining, classification2015 • data , techniques , algorithms , using , methods , based , used , approach, algorithm, method , classification, paper, learning , model, system data , using , based , model, method , algorithms , techniques , learning , methods , models, used ,algorithm, ai, system , field, ml(b) AI/ML A BSTRACTS
Table 3: Timelines of top 15 most frequently co-occuring words with “artificial intelligence“ or “machine learning” within a window size of5 in AI/ML N
EWS and AI/ML A
BSTRACTS . Words bolded in blue appear in all three years for each dataset respectively.
EWS mit’s 13.6 0.001 ai 11.7 0.007 ai 10.9 0.02robotics 12.9 0.004 algorithms 11.6 0.004 algorithms 10.6 0.003algorithms 12.5 0.004 robotics 11.5 0.004 robotics 10.5 0.004siri 12.1 0.002 azure 10.8 0.001 artificial 9.8 0.007ai 11.9 0.006 predictive 10.6 0.002 augmented 9.8 0.001
AI/ML A
BSTRACTS uci 12.6 0.002 uci 12.5 0.002 ai 10.3 0.007supervised 10.8 0.002 supervised 10.9 0.003 supervised 9.7 0.002ai 10.7 0.005 repository 10.7 0.002 classifiers 9.1 0.001repository 10.3 0.002 ai 10.4 0.003 unsupervised 8.9 0.001classifiers 9.8 0.001 classifiers 9.9 0.002 algorithms 8.9 0.009Table 4: Top five words with the highest mutual information to AI/ML over three years for AI/ML N
EWS and AI/ML A
BSTRACTS . datasets for machine learning. None of the words that appearin Table 3b consistently over time appear in the Table 4 con-sistently over time. While algorithms appear in 2019 in Table4, the rest of the words that have the highest co-occurrencefrequency with artificial intelligence and machine learning are not distinctly unique to AI/ML. In general, words withhigh mutual information to artificial intelligence and machinelearning remain consistent over time; however, few wordshave increasing mutual information, like deep and big . We measure the shift of words co-occurring with AI/MLby computing the standard deviation of the normalized co-occurrence ranks for words with frequencies in the top 1% ofAI/ML N
EWS and AI/ML A
BSTRACTS for at least one year.This measurement produces 921 results for AI/ML N
EWS and 457 results for AI/ML A
BSTRACTS . Standard devia-tion values fall between 0 (no variation) and 0.47 (maximumvariation) using this 1% frequency threshold. Table 5 dis-plays results for the standard deviation values of 0, 0.05-0.1,0.1-0.4, and 0.4-0.47 (limited to 20 words per bin) to show-case words with the least and the most variance over time (seeSupplementary Materials for full results). For the words withfluctuating co-occurrence ranks, we examine the direction oftheir shift (decreasing in rank or increasing in rank), and if https://archive.ics.uci.edu/ml/index.php a word is not observed in 2011 but is observed in 2015 and2019, we consider the word to be emerging. AI/ML N
EWS : Of the 921 resulting words fromAI/ML N
EWS , 17% of words have σ values in (0, 0.1],such as robotics and software , indicating a consistent co-occurrence frequency with AI/ML. Only two words ( siri and laboratory ) have downward trending co-occurrence ranks.Both words lose popularity from 2011 to 2015, but stay con-sistent from 2015 to 2019. The remaining words, such as ethical and quantum , have an upward trend in co-occurrenceranks. Emerging words, such as blockchain and cybersecu-rity , signal new application areas, systems, and products thatare integrating AI/ML. Words with minimal increasing ranksnot displayed in Table 5 include company names and sys-tems(e.g., ibm , watson , google , siri , and mit ) and applicationareas (e.g., biotechnology , military , and manufacturing ). AI/ML A
BSTRACTS : Of the 457 resulting words fromAI/ML A
BSTRACTS , 70% of words have σ values in (0,0.1], such as theory and statistical , indicating a consistent co-occurrence frequency with AI/ML. Three words are labeledas emerging ( convolutional , discloses , and iot ) and sevenwords ( retrieval , reasoning , genetic , web , fuzzy , cognitive ,and logic ) have minimally decreasing co-occurrence ranks.Words with increasing co-occurrence ranks signal new mod-els, systems, and techniques (e.g., adversarial , quantum , and unmaned ). ate ofChange AI/ML N EWS
AI/ML A
BSTRACTS
No shift σ = 0 advanced, algorithms, computer, data, human, information,institute, language, mining, processing, research, researchers,robotics, science, software, system, techniques, technologies,university, use analysis, classification, computational, data,engineering, information, methods, mining, model,network, neural, processing, recognition, repository,researchers, statistical, svm, technique, theory, usedMinimalincrease σ ∈ [0 . , . apps, capability, chips, competitive, cutting-edge, economic,education, government, investment, marketing, modern,monitoring, navigation, operational, quantum, revolution, risk,state-of-the-art, sensing, surveillance adversarial, analytics, apparatus, deep, equipment,obtaining, operation, quantum, rapid, relates, storage,things, utility, vehicle, voiceSignificantincrease σ ∈ [0 . , . cloud-based, defense, demand, drone, ethical, facebook,forecast, microsoft, nlp, novel, patent, policy, privacy,processors, rapid, saas, smartphones, stock, tesla, transforming big, medium, terminal, unmannedMaximumincrease σ ∈ [0 . , . apis, amazon, azure, bitcoin, blockchain, chatbots, commerce,cybersecurity, data-enabled, disruptive, ethereum, facial,flashstack, fintech, genomic, iot, newswire, selfdriving,semiconductor, startups convolutional, discloses, iotTable 5: Words (listed alphabetically) from AI/ML N EWS and AI/ML A
BSTRACTS within binned normalized rank standard deviations. Allwords with σ ∈ [0 . , . emerge in the 2015 subset of AI/ML N EWS /AI/ML A
BSTRACTS ( flashstack emerges in the 2019 subset). Generally, we find that the language surrounding AI/ML innews articles changes much more than in scientific publi-cation abstracts. By measuring the strength of word as-sociations and shifts in language over time, we find moreconsistent language use in AI/ML A
BSTRACTS than inAI/ML N
EWS (displayed in Table 4 and 5). While fre-quently co-occurring words in AI/ML A
BSTRACTS changeminimally, frequently co-occurring words in AI/ML N
EWS shift from words such as software and research to words like analytics and digital .Our word association measurements provide insight intowords that have a consistent, strong association to artificialintelligence and machine learning , as well words that havea shifting strength of association. Comparing mutual infor-mation values over time, we highlight words with strong as-sociations to AI/ML over all three years. For example, inAI/ML N
EWS , robotics and robots have consistently highmutual information values, aligning with Cave et al.’s findingthat many adults define AI in relation to robots. Words withconsistently high mutual information values in AI/ML A B - STRACTS identify commonly used models and fundamen-tal components of AI/ML, such as supervised repository ,and mining . Mutual information results from AI/ML A B - STRACTS align with Krafft et al.’s finding that most AI re-searchers define AI in terms of its capabilities and applica-tions in technical problems [Krafft et al. , 2020].Computing the standard deviation of normalized co-occurrence ranks, we highlight words that are consistent,shifting (including the shift’s direction), and emerging in text.In AI/ML N
EWS , emerging words signal new applicationareas (e.g., blockchain and bitcoin ) and words increasing inrank signal booming application areas or improved products(e.g., smartphones and chatbots ). Notably, in AI/ML N
EWS , ethical emerges in 2015, an appearance consistent with re-ports on increasing concerns in policy and society surround- ing the ethical implications of AI models [Fast and Horvitz,2017; Cave et al. , 2019; Chuan et al. , 2019]. In AI/ML A B - STRACTS , emerging words (e.g., convolutional ) highlightemerging models in AI/ML, while words increasing in rank(e.g., quantum ) highlight growing AI/ML application areas.These comprehensive results indicate how culture andtrends affect how AI/ML are perceived, applied, and definedin the context of news articles and scientific publication ab-stracts. We are able to identify words that are consistent overtime (e.g., algorithms , computers and data ), thereby demon-strating word association norms. We can also identify emerg-ing words—specifically companies, products, systems, mod-els, and technologies that have strong associations to AI/ML(e.g., facebook , quantum , and semi-conductor )—providinginsight into the evolution of AI. Artificial intelligence is challenging to study, as it is anemerging and rapidly evolving technology that is actively in-tegrated into various domains. Our work implements an au-tomated analytical approach to study the language surround-ing AI/ML over time in order to highlight consistent, shift-ing, and emerging language. We use two large-scale datasetsfrom news articles and scientific research publications, ap-plying our approach in a domain reflecting public perceptionand a domain reflecting subject-area applications. Capturingword association norms with AI/ML (e.g., robotics and algo-rithms ), as well as emerging word associations (e.g., ethical and cybersecurity ), our results not only align with prior man-ual research and surveys but also provide new insights intopublic perceptions and subject-area discussions of AI.Interesting extensions of our analysis would be to use textcorpora from different domains, such as social media text andpolicy documents as well as text in non-English languages, toprovide a global perspective of AI. eferences [Bolukbasi et al. , 2016] Tolga Bolukbasi, Kai-Wei Chang,James Y Zou, Venkatesh Saligrama, and Adam T Kalai.Man is to computer programmer as woman is to home-maker? debiasing word embeddings.
Advances in neuralinformation processing systems , 29:4349–4357, 2016.[Bryson, 2019] Joanna J Bryson. The past decade and futureof ai’s impact on society.
Towards a new enlightenment: Atranscendent decade. Madrid: Turner-BVVA , 2019.[Buchanan et al. , 2019] Erin M Buchanan, Kathrene DValentine, and Nicholas P Maxwell. English semanticfeature production norms: An extended database of 4436concepts.
Behavior research methods , 51(4):1849–1863,2019.[Caliskan et al. , 2017] Aylin Caliskan, Joanna J Bryson, andArvind Narayanan. Semantics derived automatically fromlanguage corpora contain human-like biases.
Science ,356(6334):183–186, 2017.[Cave et al. , 2018] Stephen Cave, Claire Craig, Kanta Dihal,Sarah Dillon, Jessica Montgomery, Bethany Singler, andLindsay Taylor. Portrayals and perceptions of ai and whythey matter. 2018.[Cave et al. , 2019] Stephen Cave, Kate Coughlan, and KantaDihal. ” scary robots” examining public responses to ai.In
Proceedings of the 2019 AAAI/ACM Conference on AI,Ethics, and Society , pages 331–337, 2019.[Chuan et al. , 2019] Ching-Hua Chuan, Wan-Hsiu SunnyTsai, and Su Yeon Cho. Framing artificial intelligencein american newspapers. In
Proceedings of the 2019AAAI/ACM Conference on AI, Ethics, and Society , pages339–344, 2019.[Church and Hanks, 1990] Kenneth Church and PatrickHanks. Word association norms, mutual information, andlexicography.
Computational linguistics , 16(1):22–29,1990.[Dobel et al. , 2010] Christian Dobel, Markus Jungh¨ofer,Caterina Breitenstein, Benedikt Klauke, Stefan Knecht,Christo Pantev, and Pienie Zwitserlood. New names forknown things: on the association of novel word forms withexisting semantic information.
Journal of Cognitive Neu-roscience , 22(6):1251–1261, 2010.[Fast and Horvitz, 2017] Ethan Fast and Eric Horvitz. Long-term trends in the public perception of artificial intelli-gence. In
Proceedings of the Thirty-First AAAI Conferenceon Artificial Intelligence , AAAI’17, page 963–969. AAAIPress, 2017.[G¨unther et al. , 2016] Fritz G¨unther, Carolin Dudschig, andBarbara Kaup. Predicting lexical priming effects from dis-tributional semantic similarities: A replication with exten-sion.
Frontiers in psychology , 7:1646, 2016.[James, 1890] William James.
The principles of psychology .Dover Publications, 1890.[Krafft et al. , 2020] PM Krafft, Meg Young, Michael Katell,Karen Huang, and Ghislain Bugingo. Defining ai in policy versus practice. In
Proceedings of the AAAI/ACM Confer-ence on AI, Ethics, and Society , pages 72–78, 2020.[Legg and Hutter, 2007] Shane Legg and Marcus Hutter.Universal intelligence: A definition of machine intelli-gence.
Minds and machines
Behavior research methods, instruments,& computers , 28(2):203–208, 1996.[Nelson et al. , 2004] Douglas L Nelson, Cathy L McEvoy,and Thomas A Schreiber. The university of south floridafree association, rhyme, and word fragment norms.
Be-havior Research Methods, Instruments, & Computers ,36(3):402–407, 2004.[Russell and Norvig, 2002] Stuart Russell and Peter Norvig.Artificial intelligence: a modern approach. 2002.[Sinha et al. , 2015] Arnab Sinha, Zhihong Shen, Yang Song,Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang.An overview of microsoft academic service (mas) and ap-plications. In