What's up with Privacy?: User Preferences and Privacy Concerns in Intelligent Personal Assistants
WWhat’s up with Privacy?: User Preferences and Privacy Concerns inIntelligent Personal Assistants
Lydia Manikonda, Aditya Deotale, Subbarao Kambhampati
Arizona State University { lmanikonda, adeotale, rao } @asu.edu Abstract
The recent breakthroughs in Artificial Intelligence (AI) haveallowed individuals to rely on automated systems for a varietyof reasons. Some of these systems are the currently popularvoice-enabled systems like
Echo by Amazon and
Home byGoogle that are also called as Intelligent Personal Assistants(IPAs). Though there are raising concerns about privacy andethical implications, users of these IPAs seem to continue us-ing these systems. We aim to investigate why users are con-cerned about privacy and how they are handling these con-cerns while using the IPAs. By utilizing the reviews postedonline along with the responses to a survey, this paper pro-vides a set of insights about the detected markers relatedto user interests and privacy challenges. The insights sug-gest that users of these systems irrespective of their concernsabout privacy, are generally positive in terms of utilizing IPAsin their everyday lives. However, there is a significant per-centage of users who are concerned about privacy and tookfurther actions to address the related concerns. Some percent-age of users expressed that they do not have any privacy con-cerns but when they learned about the “always listening” fea-ture of these devices, their concern about privacy increased.
We define Intelligent personal assistant (IPA) as a systemthat is capable of learning the interests and behavior ofthe user and respond accordingly. Some of these systemsthat are always on and listening in the background are be-coming ubiquitous. Despite the many possible privacy con-cerns raised by these IPAs, their popularity is continuingto soar. In the current market, there exist multiple IPAs –
Amazon Echo
GoogleHome (https://madeby.google.com/home/),
Microsoft Cor-tana
Apple Siri
Echo and Google
Home in ourinvestigation. According to a recent analysis by Adobe Dig-ital Insights , there exist 35.6 M users of voice-enabledspeakers. Where, 70.6% of them use Amazon Echo and23.8% of them use Google Home while the rest use othertypes of systems. Given the popularity of these IPAs and the https://goo.gl/nCzjNt raising concerns about privacy, now is the time to investi-gate how individuals are finding a trade-off between privacyconcerns and the convenience and efficiency of the IPAs.This paper leverages the machine learning techniquesalong with the help of human coders to investigate threemain research challenges 1) unique markers of IPAs thatmotivate individuals to use them; 2) privacy concerns as-sociated with the usage of IPAs; 3)actions taken to addressthe privacy concerns. In order to address these challenges,we investigate data about IPAs from two main sources of in-formation 1) reviews posted on the web and 2) responsesto a survey. The digital version of word-of-mouth – on-line reviews are powerful sources of information especiallythe prime factors of new product diffusion (Lin, Lee, andHorng 2011). Alongside, surveys provide an in-depth un-derstanding of the perceptions and experiences of individu-als (Rosentraub 1981). This approach of utilizing both on-line and offline sources of information ensures that the pat-terns extracted from the data provides meaningful and trust-worthy insights. Especially this provides a deeper under-standing about the privacy issues of IPAs. Summary of results:
Our analysis led to five key insightsabout the usage patterns of IPAs and how users balance pri-vacy concerns. From the online reviews – 1) Even thoughthere is a significant percentage of negativity towards IPAs,users are highly positive towards using them in their dailylives; 2) There are larger percentage of concerns with regardto privacy for Amazon
Echo predominant from January 2017compared to Google
Home . Interestingly, this is coinciden-tal to the infamous court case of an Arkansas man accusedof killing his friend in November 2015. In the trial of thiscase, the court ordered Amazon to hand over the recordingsof
Echo . Responses to the survey suggested that – 3) usersgrapple with seven different privacy issues. They are – thedevice getting hacked (68.63%), collecting personal infor-mation (16%), listening 24/7 (10%), recording private con-versations (12%), not respecting user’s privacy (6%), datastorage repository (6%), creepy nature of this device(4%);4) Even technically-savvy users are not completely aware ofthe fact that these devices are always listening. Once theythey were informed of this fact, their concernes about pri-vacy increased; 5) Users who mute their microphone ex-pressed privacy as one of their main reason to mute the de-vice. a r X i v : . [ c s . C Y ] N ov est Buy Google Search AmazonAmazon Echo +ve (26,402) 4880 912 20,610 Amazon Echo -ve (4,833) 256 67 4510
Google Home +ve (5,495) 4638 857 0
Google Home -ve (655) 514 141 0Table 1: Distribution of the reviews across the two IPAsThe results of our studies thus shed some light on the com-plex ways users are still wrestling with the tension betweenthe utility of the IPAs and the privacy concerns they raise.We hope these can help in some small way towards betterdesign of IPAs.
The reports shared by Business Insider states that 35.6Musers in the United States use voice-enabled speakers whichinclude popular IPAs – Amazon
Echo and Google
Home .Due to the popularity of these products, there are multiplesources of information about the features of these products,articles about comparing the products, reviews written on-line by the users of these products. We wish to utilize thewealth of information available online in the form of prod-uct reviews to investigate the privacy aspects of IPAs. Thefirst task is to crawl the reviews and process them to createthe final dataset used in this investigation. We crawl the re-views from 1) Google Search, 2) Amazon ( amazon.com )and 3) Best Buy ( bestbuy.com ). We obtain a total (in-cludes both positive as well as negative reviews) of 31,235reviews for Amazon Echo and 6,150 reviews for GoogleHome. For each review we crawled, there are multiple at-tributes attached with it – rating, title, author, date posted,review text, Echo or Google
Home ) and different factors associated withtheir interest in using this product. This set of questions arefollowed by questions about their concerns about privacy orsecurity aspects along with their opinion on muting the mi-crophone. At this section of the survey, the participants are Python crawler https://docs.python.org/2/howto/urllib2.html aware of the default setting of always on and listening con-tinuously for the trigger phrase. We then repeat the questionsabout their concerns about privacy and end the survey withan open-ended question about any feedback about chang-ing the product to respect user’s privacy. Two researchersread the responses to these open-ended questions posed inthe survey and coded the data so as to answer the questionswe posed earlier. This coding process was done separatelyand until both the researchers reached an agreement.
Age
49 18-20, 2 21-24, 0 25-30, 0 30or more
Gender
45 Male, 6 Female
Main usage
Listening to Music(44), PersonalAssistant(24), Controlling otherdevices(11), Getting Informationlike Weather(36)Table 2: Survey Demographics and InformationIrrespective of working with the crawled online reviewsdataset or the responses to the survey, the data should bepre-processed due to the considerable noise in terms of non-english characters, emoticons, informal language, etc. (c.f.(Baldwin et al. 2013)). Without the processing of the data,the machine learning algorithms that are utilized in the lat-ter part of the analysis are forced to work with a data that isunstructured and ambiguous. Due to these reasons the datais processed in such a way that the emoticons, non-englishcharacters are removed and structure is imposed on this data.Also, since the reviews are crawled from multiple onlinesources, there is a possibility that we might’ve crawled cer-tain reviews multiple times.
We utilize the online reviews crawled for both Amazon
Echo and Google
Home in such a way that the positive reviewsfor both these devices are merged together in this section ofthe analysis unless specified. Similarly, the negative reviewsare merged together for both the devices. We consider the4-star rating and the 5-star rating as positive reviews and 1-star, 2-star and 3-star ratings as negative reviews. Table 1shows that the number of ratings for negative reviews arefewer than the positive reviews however, the average lengthof negative reviews is larger than the positive reviews.
Online expressions in the forms of reviews and ratings, arethe artificial currencies for marketers to learn about the im-
Rating Score vs Number of Reviews
Amazon Echo Google Home (a)
Rating Score vs Avg. Length of a review
Amazon Echo Google Home (b)
Figure 1: (a)Rating score vs the number of reviews for the corresponding score; (b) Rating score vs average length of thereviews for the corresponding scorepact of their products on society. Detecting emotions fromthese reviews help shed light on the different aspects ofIPAs. Specifically in the context of this paper, emotionscould comprehend the interactions with IPAs especiallytheir effect on users and their cognitive abilities. To detectemotions, we employ the psycholinguistic lexicon LIWC(http://liwc.wpengine.com/) on the text associated with thereviews we crawled. It considers the following 10 emotionalattributes motivated from prior work on the public opinionsabout AI (Manikonda, Dudley, and Kambhampati 2017).The emotional attributes are: insight , sad , anger , home , neg-ative emotion , family , cognitive mechanisms , bio , positiveemotion and sexual .Figure 2: 10 prominent emotional attributes extracted usingLIWCFigure 2 shows that since systems are intelligent in na-ture, it is interesting to observe that they are engaging thecustomers of these products in terms of their cognitive abil-ities ( cogmech =86.88% (+ve) and 89.76%(-ve)). Cognitivemechanisms can be described as a richer way of reason-ing (Tausczik and Pennebaker 2010). Negative reviews havelarger percentage of insight words that may suggest the ac-tive reassessment of a theory. Using a large number of in-sight words especially in the negative reviews may suggestthat customers focus on providing a detailed in-depth re-views justifying their ratings. The average length of the re-views shown in Figure 1 support this observation that thelength of negative reviews are longer than the positive re-views for the IPAs considered in this study. There is a sig-nificant percentage of words that belong to the categories of home , family , bio and sexual which focus on words relatedto different aspects of home, family, themselves and gen-der respectively. This suggests that users describe how theirfamily or they evaluate this product. For example, one of thereviewer posts – “My wife thought it was fun [...]”, “My kidslove the Pandora connection [...]” .Emotions related to negativity – sad , anger , negemo arepresent in even the positive reviews apart from the negativereviews as shown in Figure 2. Irrespective of negativity to-wards these products, they do have a higher percentages ofpositivity ( posemo ). For example, a negative review states– “ Kind of neat, kind of cool but limited capability withGoogle play music. Bluetooth is nice but the range limita-tions make the Echo less than useful[...] ”. On the other hand,users who are giving high rating also expressed concernsabout privacy. For example a reviewer who gave a 4-star rat-ing said – “
Works decent for voice queries, great speaker,the reason it isn’t 5 stars is the privacy concerns, since it isalways listening ”. In order to determine how and why users are utilizing IPAsin their everyday lives, discovering the latent topics focusedby their reviews could be very helpful. Topics are primar-ily useful in terms of understanding the underlying semanticstructure in the reviews. Even though emotion analysis pro-vides a clear view on the attitudes of individuals towardsIPAs, they do not necessarily specify the reasons for theseemotions. These reasons can be captured by extracting thesemantic latent topics.To conduct a latent topic detection, we utilize the popularLatent Dirichlet Allocation algorithm (Blei, Ng, and Jordan2003) that mines the latent topics given a set of documents.We extract the top-4 latent topics from both positive and neg-ative reviews separately. Once the topics are extracted, cod-ing process is conducted by 2 human encoders to summa-rize the topics (summarized topics in Table 3). These topicsextracted describe the reasons why users either like or dis-like the product. For example, topics extracted from negativereviews are about how these IPAs have limited understand-ing of questions posed by the users along with the technical eview Type Summarized Topics and their distributionsPositive Reviews
Query search about news, weather, calendar, etc (31.6%);Voice recognition and speaker capabilities (8.3%); pur-chased or got it as a gift and how the family has fun play-ing with it (34.4%); Easy integration with other devicesor playlists (25.8 %)
Negative Reviews
Limited search features (63.8%); wifi and network con-nection issues (14.2%); contacted customer service forreplacement or warranty and waste of money (22.0%)Table 3: Topics extracted from the positive reviews and negative reviewsfaults of the devices. These definitely point to the fact thatthe speech recognition, relevancy of answers for the ques-tions are definitely drawing negative criticism from usersespecially for Amazon
Echo not so much for Google
Home .On the positive note, the easy setup of IPAs, smart controlof lights, ease of communicating with speech are key factorsthat are attractive to the users.
A traditional n -gram analysis cannot provide us with anaccurate information about the semantic distances betweenwords but are totally relied on the occurrence frequency. Co-occurrence or semantic proximity on the other hand is de-fined as the concept when two terms from a corpus co-occurin a certain order. In computer vision, co-occurrence matrixis utilized to measure the texture of an image that includescolor and intensity. Leveraging this idea, co-occurrence ma-trix in a linguistic context is measured between pairs ofwords to extract the distinct markers of the IPAs – Ama-zon Echo and Google
Home . These words are representedin the form of vectors by the Word2Vec model. Given theadvantages of Word2Vec models (Mikolov et al. 2013), weenvision that measuring the co-occurrence matrix in termsof the Word2Vec space will identify distinguishable mark-ers between the IPAs that emotion analysis and topic analy-sis might not have discovered. Even though the performanceof these approaches increase as the amount of training data(which is the set of reviews for a given category) increases,they detect meaningful co-occurrence patterns that are bothsemantically and syntactically related.Table 4 lists the top co-occurring words with the phrase”privacy issues” that are syntactically and semantically re-lated. Based on these top co-occurring words shown in thistable, Amazon
Echo received more concerns about privacythan Google
Home . One reason could be that
Echo hasbeen existing since 2015 where as
Home was launched in2016. The other reason could be the negative attention re-ceived by
Echo during the infamous court case in December2016 (https://goo.gl/nmrF8J). We also speculate that sinceGoogle already has an extended presence in terms of pro-viding email services, mobile operating system, etc., usersmight be comfortable even if
Home is always on. As de-scribed in the earlier section, positive reviews contain nega-tive comments which we also see in this particular context.The words that are highly associated with “privacy issues”extracted from the positive reviews are very negative. Es-pecially, the words disappointed , lack , trouble , hate repre- sent a deeper level of frustration towards this product. Whenit comes to the words extracted from the negative reviews,some of the users are either unplugging the device or re-turning the product. On the other hand, Google Home usersassociated the word “returned” with “privacy issues”.
Amazon Echo Google Home +ve Reviews -ve Reviews +ve Reviews -ve Reviews lack failure noise returnedserious security problemconversation seriousdisappointed unpluggedproblem stuckunplug lackissue returningtrouble sadlyhate
Table 4: Top co-occurring words with the phrase “privacy is-sues”; Co-occurrence patterns are extracted using Word2vecmodel trained on the reviews posted online.
Temporal Trends of Mentions about Privacy:
In thisage of Twitter acting as a news media platform, individu-als are gaining better opportunities to get exposed to person-alized news about real world scenarios. Especially, due tothe rising fears of machines taking over the world, scenarioswhere intelligent systems are playing a key role in the so-ciety are gaining more attention from the media as well aspublic. By utilizing the review ratings, we conduct a tempo-ral analysis to identify interesting temporal trends (shown inFigure 3). We noticed that since December 2016, there havebeen concerns about the aspects of privacy predominantlyamong the negative reviews. This pattern is more evident forAmazon
Echo . Interestingly, it is coincidental with the infa-mous court case in December 2016 where the court orderedAmazon to hand over the information recorded by Echo.Some of the reviews after December 2016 are: “I returnedmy Echo because of privacy concerns. Now judges are issu-ing subpoenas in court cases for Echo records [...]” written inMay 2017, “[...]with the recent court revelations [...] A legalnightmare potential for anyone who values at least a mod-icum of civil rights or cares about not only their privacy, butany of their visitors as well” written in January 2017, “[...]very cool to most, but for me the scope of privacy intrusionis beyond anything I’ve even imagined. Alexa, I think I can https://goo.gl/nmrF8Ja) Privacy trends for Echo -ve Reviews (b) Privacy trends for Echo +ve Reviews Figure 3: Privacy Trends over Time for Amazon
Echo manage well without you.” written in March 2017. This maysuggest the key role played by the news stories and movieson affecting the user concerns about privacy.
51 participants were participated in the survey where theywere asked a series of open-ended questions to understandthe usability patterns and privacy concerns associated withthe usage of these products. The demographics of the surveyare shown in Table 2. This set of participants comprises of45 males and 6 females who are mostly from the age groupof 18-20 years old. Except 1 participant, rest of the other arecurrently using Amazon
Echo at home. We summarize thekey takeaways about privacy concerns from the responses tothe survey.
Echo in their daily lives. Each participant can list oneor more concerns about the privacy. Two researchers inde-pendently aggregated these responses and coded them intoseven categories of privacy concerns.
1. Hacking the device:
16% survey respondents men-tioned that hacking or tampering their device through remoteweb attacks are very concerning. For some, hackers interfer-ing with their device was the first concern they thought of:“I hope hackers don’t have access to my info[...]’. Othersare concerned that they might be vulnerable to malware orcybersecurity breaches: “[...] information can be interceptedby 3rd parties such as malware, cybersecurity breaches, etc”.
2. Collection of Personal Information:
16% of our sur-vey participants mentioned that collection of their personalinformation by the device is a primary privacy concern.For most of these participants the way that the device col-lects their personal information including details about theircredit card information are concerning. Some of the example responses are: “ collection of personal information ”, “ [...]Iam more concerned about saying my credit card informa-tion ”.
3. Recording Private Conversations:
12% of our sur-vey participants indicated that they are concerned about thedevice recording their private conversations. Some of theresponses are: “Maybe discussing something private andsomeone is listening”, “I do not want it to record my con-versations”, “unknown individuals could be listening to mylife and that is totally appalling”.
4. Listening 24/7:
10% of survey respondents expressedtheir concern that this device is listening to them all thetime. Some of the responses are: “It listens to my life 24/7”,“Knowing every little thing about me and someone con-stantly watching”. Few respondents mentioned that they arenot very popular so they don’t mind the device listening tothem but there are times at which they don’t want the deviceto keep on listening: “I am not someone who is super wellknown but some times I just don’t want the device listeningto me”.
5. Respecting the user’s privacy:
6% of survey partici-pants mentioned that the device should respect their privacy.They didn’t specify much in details but they said their pri-vacy should be of utmost consideration. Some of their re-sponses are: “[...] I know how important security is and Ivalue being able to keep my information private”.
6. Data Storage Repository:
6% survey respondents werecurious about where and how their information is beingstored and utilized by the company that manufactured thisdevice. Some of their responses are: “where they keep theinformation [...]”, “need to know where my history and pri-vate information is stored”.
7. Creepy nature:
4% of our survey participants issuedconcerns about the creepiness of this device. Some of theirresponses are: “Its a bit creepy [...]”, “I do not like the factthat it can be used as evidence”. .2 When users learned about the devices alwayslistening, their privacy concerns increased
The survey asks the participants to rate their level of concernon a 1-to-5 likert scale before and after they were told aboutthe “always on” feature to measure their change of concernsabout privacy. First, the users were asked ( before score ) torate their level of concern. This question was followed by an-other question about their awareness of the fact that the de-vice is always on and listening. 19.6% of participants men-tioned that they are not aware of this fact and then they wereinformed about this listening aspect of the device. The userswere then asked ( after score ) to rate their level of concernafter knowing this additional information. We analyzed howthe rate of concern for the 19.6% of participants who arenot aware of this fact about the device. To our surprise therate of concern went up high by 50% of the respondents whomentioned that they do not know the device is always on andlistening.We dug a little deep into the responses of users whosescores were increased (for example, 1 to 3 and 2 to 5) es-pecially the other factors of awareness about the product.These participants also mentioned that they are not awareof the available option that they can mute the microphone.Some of these users also suggested that factors like thisshould be specified to the users upfront. This shows that thetechnically-savvy users are not completely aware of the factthat these devices are always listening but got alarmed afterlearning about this fact.
Many survey respondents expressed their concerns that thedevice is constantly listening to their conversations 24/7.47% of the respondents mentioned that they mute their mi-crophone on this device and 29.4% of them do not mutethe microphone. The remaining 23.6% survey respondentsdo not know that the option of muting their microphone isavailable. Then we asked the respondents their motivation tomute their microphones. Their aggregated responses suggestthese reasons below: • background conversations • when not in use • privacy concerns (for example, someone responded that“watched Snowden movie and made me weary of thisstuff” ) • responds to questions when not needed Intelligent systems are becoming part of our lives as moreand more individuals are relying on them for driving (Kimet al. 2013), work management (Ferguson, Allen, and oth-ers 1998; Myers et al. 2007), time scheduling (Zhang et al.2013; Manikonda et al. 2014), information and email or-ganization (Mitchell et al. 1994; Dumais et al. 2016), andresponding to factual questions (Brill, Dumais, and Banko2002; Ravichandran and Hovy 2002). These systems lever-age AI to learn and reduce the mistakes made by users due to cognitive overload (Myers et al. 2007). In this context,there are multiple debates about whether AI that is primar-ily used by these intelligent systems is a blessing or curse tothe society. One of our previous works (Manikonda, Dud-ley, and Kambhampati 2017) investigated the public per-ceptions of individuals about AI. These individuals com-prise both AI experts and common users on Twitter (a popu-lar micro-blogging platform) who are sharing their personalopinions and information about AI. However, we believethat researching the interactions of individuals with the intel-ligent systems has greater potential to understand the impactof AI on the society to a certain level especially the privacyconcerns that come along with that (Czibula et al. 2009).The digital version of word-of-mouth – online reviewsare becoming the major source of information especiallya prime factor of new product diffusion for potential buy-ers (Lin, Lee, and Horng 2011). Existing literature (Park,Lee, and Han 2007; Lee 2009) focusing on the impact of re-views state that better reviews led to improved sales of prod-ucts. Also, surveys provide an in-depth understanding of theperceptions and experiences of users (Rosentraub 1981). Inspite of the fact that the aspects of privacy with respect to in-telligent systems are becoming popular, it has attracted rela-tively less attention from the research community. In this pa-per, using the online web reviews and a survey, we presenta qualitative and quantitative analysis to study the privacyaspects of IPAs.
We used online reviews along with the survey responsesfrom the users of IPAs to draw the following conclusions:1) People are overall very positive about these devices andcomfortable to use them; 2) However, there are multiple pri-vacy issues; 3) Many users expressed that they are aware ofthe privacy concerns and they do take actions: a) returningthe product; b) muting the microphone; c) limiting the us-age – example only using to set alarms; 4) Even technically-savvy users (from our survey) are not completely aware ofthe fact that these devices are always on and listening. Oncethey learned that these devices are always on and listening,their rate of concern increased; 5) Users who mute their mi-crophone expressed privacy as one of their main reasons tomute the device; 6) The temporal trends suggest that maybe the media, in the form of news and movies are makingusers aware of the privacy concerns associated with thesedevices. Most of the respondents suggested that the privacyissues can be addressed by making these devices transparent.We hope that the results of our investigation highlights someof the complex ways users are struggling to find a balancebetween using these devices and the associated privacy con-cerns. We hope that these insights could help better designthe IPAs with more transparency.
References [Baldwin et al. 2013] Baldwin, T.; Cook, P.; Lui, M.;MacKinlay, A.; and Wang, L. 2013. How noisy social mediatext, how diffrnt social media sources? In
IJCNLP , 356–364.Blei, Ng, and Jordan 2003] Blei, D. M.; Ng, A. Y.; and Jor-dan, M. I. 2003. Latent dirichlet allocation.
Journal ofMachine Learning Research
Proceedings of EMNLP . Associationfor Computational Linguistics.[Czibula et al. 2009] Czibula, G.; Guran, A.-M.; Czibula,I. G.; and Cojocar, G. S. 2009. Ipa-an intelligent per-sonal assistant agent for task performance support. In
In-telligent Computer Communication and Processing, 2009.ICCP 2009. IEEE 5th International Conference on , 31–34.IEEE.[Dumais et al. 2016] Dumais, S.; Cutrell, E.; Cadiz, J. J.;Jancke, G.; Sarin, R.; and Robbins, D. C. 2016. Stuff i’veseen: a system for personal information retrieval and re-use.In
ACM SIGIR Forum , volume 49, 28–35. ACM.[Ferguson, Allen, and others 1998] Ferguson, G.; Allen, J.;et al. 1998. Trips: An integrated intelligent problem-solvingassistant. In
Proceedings of the National Conference on Ar-tificial Intelligence , 567–573. JOHN WILEY & SONS LTD.[Kim et al. 2013] Kim, J.; Kim, H.; Lakshmanan, K.; and Ra-jkumar, R. R. 2013. Parallel scheduling for cyber-physicalsystems: Analysis and case study on a self-driving car. In
Proceedings of the ACM/IEEE 4th International Conferenceon Cyber-Physical Systems , 31–40. ACM.[Lee 2009] Lee, S.-H. 2009. How do online reviews affectpurchasing intention?
African Journal of Business Manage-ment
Social Behavior and Personality: an international journal
Second AAAI Conference onHuman Computation and Crowdsourcing .[Manikonda, Dudley, and Kambhampati 2017] Manikonda,L.; Dudley, C.; and Kambhampati, S. 2017. TweetingAI: perceptions of ai-tweeters (AIT) vs expert ai-tweeters(EAIT).
CoRR abs/1704.08389.[Mikolov et al. 2013] Mikolov, T.; Chen, K.; Corrado, G.;and Dean, J. 2013. Efficient estimation of word representa-tions in vector space.
CoRR abs/1301.3781.[Mitchell et al. 1994] Mitchell, T. M.; Caruana, R.; Freitag,D.; McDermott, J.; and Zabowski, D. 1994. Experiencewith a learning personal assistant.
Commun. ACM
AI Magazine
International journal of electronic commerce
Proceedings of the 40th annualmeeting on association for computational linguistics ,41–47.[Rosentraub 1981] Rosentraub, M. S. 1981. The use of sur-veys of satisfaction for evaluations.
Policy Studies Journal
Journal oflanguage and social psychology