IntentsKB: A Knowledge Base of Entity-Oriented Search Intents
IIntentsKB: A Knowledge Base of Entity-Oriented Search Intents
Darío Garigliotti
University of [email protected]
Krisztian Balog
University of [email protected]
ABSTRACT
We address the problem of constructing a knowledge base of entity-oriented search intents. Search intents are defined on the level ofentity types, each comprising of a high-level intent category (prop-erty, website, service, or other), along with a cluster of query termsused to express that intent. These machine-readable statementscan be leveraged in various applications, e.g., for generating entitycards or query recommendations. By structuring service-orientedsearch intents, we take one step towards making entities actionable.The main contribution of this paper is a pipeline of componentswe develop to construct a knowledge base of entity intents. Weevaluate performance both component-wise and end-to-end, anddemonstrate that our approach is able to generate high-quality data.
CCS CONCEPTS • Information systems → Query intent ; Information extraction ; KEYWORDS
Knowledge base construction; search query intents; entity types
ACM Reference Format:
Darío Garigliotti and Krisztian Balog. 2018. IntentsKB: A Knowledge Baseof Entity-Oriented Search Intents. In
The 27th ACM International Conferenceon Information and Knowledge Management (CIKM ’18), October 22–26, 2018,Torino, Italy.
ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3269206.3269257
Many information needs behind people’s searches revolve aroundspecific entities. Entities, such as people, organizations, or locationsare natural units for organizing information; they can provide notonly more focused responses, but often immediate answers [8].Alongside mere informative exploration, users frequently look fortransaction-oriented entity intents, like booking a flight or a hotel.In this paper, we propose to build a knowledge base (KB) of entity-oriented search intents. Specifically, we identify the main searchintents for a representative sample of entity types, and representthem in a structured fashion. These machine-readable statementscan be used for automatic querying and reasoning about searchintents, and represent one step towards making entities actionable .Most entity-oriented queries consist of an entity name, comple-mented with context terms to express the underlying intent of the
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected].
CIKM ’18, October 22–26, 2018, Torino, Italy © 2018 Association for Computing Machinery.ACM ISBN 978-1-4503-6014-2/18/10...$15.00https://doi.org/10.1145/3269206.3269257
Table 1: An excerpt from our knowledge base, for IntentID:
We propose to represent search intents at the level of entity types,where a type is a semantic class that groups multiple entities. Thisallows us to capture intents common to many entities, resulting ina representation that is space-efficient and generalizes well to long-tail and emerging entities [4, 7]. We present a pipeline approach thatconsists of four main components. First, we acquire entity-bearingqueries of given types and aggregate refiners to obtain type-levelquery patterns (e.g., “ [hotel] booking " or “ [airline] customer service ").Second, we classify each of these type-level refiners into four main intent categories , property, website, service, or other, based on howthat information need can be fulfilled. Third, type-level refiners thatexpress the same underlying intent are clustered together. A clusterof refiners corresponds to the various ways of expressing that in-tent in an actual query (like “ [hotel] booking ," “ [hotel] reservations ,"“ [hotel] book a room ,” etc.). Fourth, structured representations ofintents are created by assigning each intent a unique intentID. Thisstructured representation then contains the type of the entity, thecategory of the intent (according to the four main intent categories),and the different ways that intent may be expressed. Additionally,each piece of information is assigned a confidence score. Table 1shows an excerpt from our IntentsKB, which represents the intentof customer service for an airline.In summary, this paper makes the following contributions: • We formally define the problem of constructing a knowledgebase of entity-oriented search intents, and design a model forstructured representation of this knowledge (Sec. 3). • We propose a pipeline framework to build the knowledge base,consisting of refiner acquisition, refiner categorization, intentdiscovery, and knowledge base construction stages (Sec. 4). • We evaluate the components of our pipeline against editorialjudgments, and, using a small seed of labeled data, generate aknowledge base comprising over 30k intents (Sec. 5).The resulting
IntentsKB knowledge base and the correspondingresources are made available at http://bit.ly/cikm2018-intentsKB.
Most related to our paper is the work by Reinanda et al. [9], whoexplore entity aspects in user interaction log data. Beyond findingaspects by comparing clustering methods over refiners, they address a r X i v : . [ c s . I R ] S e p he tasks of ranking such intents for a given entity independentlyfrom a query and recommending aspects. Unlike them, we (i) workon individual query refiners, (ii) model entity intents at the levelof types, (iii) consider always a query context of entities, (iv) useapproaches defined entirely in absence of log data, and (v) obtainuniquely identified intents. Actions are a particular kind of entity intent, which we representusing the service category. The schema.org ontology is equippedwith a dedicated actions vocabulary. We find this representationwell-oriented to model actions, yet not expressive enough regardingthe actual ways of expressing the intent in a query. Also, the currentschema.org model does not allow to represent actions at the entitytype level. Lin et al. [4] propose the model of active objects as arepresentation of an entity associated with a ranked list of potentialactions. These actions are simple surface forms mined from querylogs. While our envisaged scenario is the same, we differ from themin that we (i) model the spectrum of actions at the level of entitytypes, (ii) do not work with a fixed set of mined actions but ratherdiscover them automatically from query intents, and (iii) obtainstructured triples with uniquely identified actions and their surfaceforms. The action mining task at the NTCIR Actionable KnowledgeGraph track [1] addresses the problem of ranking potential actionsfor a given entity of a given type. Here, an action comprises of averb and possibly an object or modifier. While this effort points inthe same direction as our work, we note the following limitations:(i) they consider entity type only as input signals, but the outputactions are required at the entity level, (ii) they do not account foractions uniquely identified in a KB, and (iii) they force an action tobe expressed as a verbal phrase, which is not the case for most ofthe service-oriented intents we observed. Our goal is to build a knowledge base of entity-oriented searchintents ( intents for short), at the level of entity types. Each searchintent is uniquely identified by an intentID and is described by an intent profile . The KB consists of a set of ( subject , predicate , object , confidence ) quadruples. We formally define the KB as a relationalknowledge representation model: IntentsKB = Q T ∪ Q C ∪ Q L ,where the three sets of quadruples are partitioned as follows: • Q T ⊆ I × { searchedForType } × TY PES × [ , ] ; • Q C ⊆ I × { ofCategory } × CAT EGORIES × [ , ] ; • Q L ⊆ I × { expressedBy } × ST RI NGS × [ , ] .The set I consists of the unique intent identifiers. searchedForType,ofCategory, and expressedBy are predicates used for associatingan intent with an entity type, intent category, and possible lexi-calizations, respectively. TY PES is the set of entity types in thereference KB, and
CAT EGORIES is a scheme of intent categories(that is, { Property, Website, Service, Other } , cf. Sect. 4.2). Figure 1 displays an overview of our framework for constructing aknowledge base of entity-oriented search intents. As an example of this last item, “hilton taxi” very likely expresses the intent of gettinga taxi to the hotel without being the refiner “taxi” a verbal phrase.
In this first stage, we obtain popular type-level query patterns. Wesample queries for a set of prominent entities, extract refiners fromthem, and aggregate these refiners across entity types. Specifically,we focus on refiners that complement a mention of a particularentity, i.e., all queries follow the pattern “ [entity] [refiner] ,” wheresquare brackets indicate placeholders. The entity mention is thenreplaced with the corresponding entity type in the reference knowl-edge base. For example, given the queries “ sydney map ” and “ parismap ,” we extract the type-level query pattern “ [travel destination]map .” Hereinafter, type-level refiner , or simply refiner , refers to thequery suffix in a type-level query pattern.
In recent work, we have proposed a suitable scheme to classifyentity-related search intents [3]. This scheme consists of four intentcategories , which focus on the way and the type of source to fulfillthe information need. • Property : The refiner is about getting a specific entity propertyor attribute that can be looked up in a knowledge base. For exam-ple, “children” in the query “ angelina jolie children ,” or “openingtimes” in “ at&t stadium opening times .” The criterion does notrequire the refiner to exist as a property in an actual knowledgebase, but rather its existence to be reasonable. • Website : The refiner looks to reach a specific website or applica-tion, which is a rough equivalent of navigational queries in [2].For example, “twitter” in the query “ karpathy twitter .” • Service : The refiner expresses the need to interact with a service,possibly by redirecting to an external site or app. For example,“menu” in the query “ keens steakhouse menu ” would indicate theneed to access to an external site for reading the restaurant’smenu. As another example, “new album” in “ eric clapton newalbum ” looks for a service to read about, or listen to, or buythe new album. The interaction would possibly involve furtherparameters, like “from” and “to” values for “ticket price” in thequery “ jpass bullet train ticket price .” • Other : None of the previous ones is applicable. For example, “jr”in the query “ tim hardaway jr ” merely serves to disambiguatethe person from other people with the same name.While this categorization comes from our prior work, the automaticassignment of intent categories to refiners has not been addressedto date. We approach this task, referred to as refiner categorization ,as a single-class classification problem using supervised learning.Each type-refiner pair defines an instance, and the four categoriesdefined above are the possible classes.
We compute attributes at the lexical levelof a refiner. Many of these signals are detected in the search resultsobtained from a major Web search engine. For every type-refinerpair, we take the most prominent entity assigned to that type (seeSect. 5.1 for how entity prominence is measured). We then searchfor the corresponding actual entity-bearing query. For example, forthe type-level query “ [travel destination] map ,” the correspondingentity-bearing query is “united states map.” We exploit the top10 search results to quantify, among others, the size of the set oftheir URL domains, or, following Reinanda et al. [9], the averageJaro distance between a refiner and each result URL. The detailed efinerAcquisition RefinerCategorization IntentDiscovery Knowledge BaseConstruction [hotel] booking[hotel] deals[hotel] from airport[hotel] address… WebsiteServiceServiceProperty…
Hotel_BookingHotel_BookingHotel_BookingHotel_ArrivingHotel_ArrivingHotel_Arriving… ofCategoryExpressedByExpressedByofCategoryExpressedByExpressedBy…
Service"rooms""reserve"Service"from airport""taxi"… conf conf conf conf conf conf …from airporttaxi… Hotel_Arriving roomsmake a reservation…
Hotel_Booking[hotel] booking[hotel] deals[hotel] from airport[hotel] address… savoy hotel bookingsavoy hotel address…savoy hotel sheraton dealssheraton from airport…sheraton
Figure 1: Overview of our proposed framework. Top: pipeline architecture. Bottom: an example through all its stages. description of the full set of lexical features can be found in theonline repository accompanying this paper.
The semantic similarity between a refiner r and a type t is defined as the cosine similarity cos ( r , t ) betweenthe centroid word embedding vectors of the refiner terms ( r ) andtype terms ( t ). This measure captures the compositional natureof words in both the type label and the refiner, which was shownin [6] to be an effective attribute of a phrase. We use pre-trainedword embeddings provided by the word2vec toolkit [5]. Once each type-level refiner is mapped to a category as describedabove, we proceed to discover the intents underlying those refiners.We achieve this by clustering the refiners that express the same userintent. We make use of the intent categories that have been assignedin the previous step, that is, two refiners in the same cluster musthave the same intent category. Following [9], we apply hierarchicalagglomerative clustering (HAC) over the distributional semanticspace of the refiners for each type. As before, we use pre-trained word2vec word embeddings. For each refiner, we take its vector ifthe refiner is in the embedding vocabulary, otherwise we assign itthe normalized centroid of the vectors of its terms.The clusters merging step of HAC is stopped when all the inter-cluster distances are above a certain cut-off threshold. This thresh-old, γ c , t , is chosen for each combination of intent category c andentity type t . However, to avoid overfitting, we only learn a singleparameter ϵ c for each intent category, and then set γ c , t = ϵ c M c , t ,where M c , t is the maximum distance between any pair of refinersbelonging to intent category c and type t . This may be seen as away of normalization, to account for various cluster sizes. To findthe best ϵ c value, we perform a grid search over [ .. ] and pick thevalue that maximizes the evaluation score against the ground truthclusters, available as training data. In the last step, we construct the full knowledge base representationof intents, i.e., create intent profiles . Let i denote an intent profile.It consists of the set of refiners R ( i ) that were clustered together.Recall that all these refiners have previously been assigned the sameintent category, therefore, there exists a single intent category forthe profile. The profile is assigned a unique intentID . In the interestof readability, it is a concatenation of the entity type, a numerical ID,and the label of the refiner which is closest to the intent centroid.Recall, that each intent profile has three types of predicates. Wedefine the confidence for each as follows: • searchedForType: since entity type information comes from acurated KB, the confidence in the assigned type is always 1. • ofCategory: we take the average categorization confidence ofthe refiners in the profile: | R ( i )| (cid:205) r ∈ R ( i ) α ( r ) , where α ( r ) is theassociated confidence score from the intent categorization step. • expressedBy: it is the similarity between the refiner r and thecentroid of all refiners in R ( i ) . In our case, refiners are repre-sented as embedding vectors and cosine is used to measure theirsimilarity (cf. Sect. 4.2).We also assign a confidence score to the intent profile itself, bytaking a linear mixture of the category confidence α ( c ) and theaverage refiner confidence α ( r ) : α ( i ) = (cid:16) α ( c ) + | R ( i )| (cid:213) r ∈ R ( i ) α ( r ) (cid:17) . (1) This section presents the evaluation of our approach.
We use the type system of Freebase. It is a two-layer categorizationsystem, where types on the leaf level are grouped under high-leveldomains. We focus on popular DBpedia entities to benefit froma larger and more representative selection of information needs.Entity popularity is measured by the number of times the entity’sEnglish Wikipedia article has been requested. We set empiricallya popularity threshold of 3,000 page views per article over a spanof one year (from June 2015 to May 2016). Given a Freebase type,we select it if it covers at least 100 entities with a popularity abovethe threshold. There are a total of 634 such types.In a second step, we collect query suggestions from the GoogleSuggestions API for top 1,000 most popular entities per type. Then,we obtain type-level query refiners by replacing the entity by itstype, as described in Sect. 4.1. Finally, we retain only those refinersthat occur in at least 5 suggestions for the given type. This leads toa total of 63,148 distinct type-level refiners for 631 types.
Our dataset consists of 4,490 instances labelled with one of thefour intent categories. For obtaining the search results exploited bymany features, we utilize the Google Search API. In a preliminaryround, we experimented with a variety of classifiers. Results arereported only for the best performing classifier, which is RandomForests. We used the following parameters: the number of trees Provided by Wikistats: https://dumps.wikimedia.org/other/pagecounts-ez/. able 2: Refiner categorization results.
Feature Set AccuracyFeature group I (lexical) 0.5920Feature group II (semantic)
Combined (I + II) 0.6150
Table 3: Clustering refiners using oracle vs. automatically as-signed intent categories, measured in terms of homogeneity,completeness, and V-measure scores.
Category Hom. Compl. V-measureOracle categories 0.9494 0.7270 0.8201Automatic categorization 0.8872 0.6872 0.7710 is 100 and the maximum depth of the trees is approximately thesquare root of the feature set used in that setting. We train themodel using five-fold cross-validation, and report the results inTable 2. We find that semantic features perform better than lexicalfeatures. When combining the two, the resulting performance isinferior to using semantic features alone. Therefore, we only theuse semantic feature group in the remainder of our experiments.
We evaluate the clustering approach in two evaluation settings: (i)an “oracle” setting, which uses the ground truth intent categories,and (ii) a realistic setting, where the categories are automaticallyassigned by our refiner categorization component. We perform five-fold cross-validation for each evaluation instance, with the samepartition of folds used for evaluating refiner categorization. Thetraining folds are used to optimize ϵ c for each category as describedin Sect. 4.3. The clusters obtained for a given type are “flattened” byignoring the intent categories, and are evaluated against the groundtruth clusters flattened in the same way. This serves to eliminate,to some extent, errors that originate from incorrect refiner catego-rization. Table 3 presents the results. We find that our automaticsetting can achieve a V-measure that is only about 6% lower thanusing oracle categories. In this last part, we evaluate the end-to-end approach by applyingthe pipeline, which we trained on 50 types (cf. Sect. 5.1), on theremaining 581 types. As a result, we obtain 31,724 intent profilesthat comprise a total of 155,967 quadruples. We proceed to estimateits expected overall quality by taking a stratified sample from thegenerated quadruples. Considering the confidence scores associatedwith the profiles (cf. Eq. (1)), we partition the range [0..1] into 5equally-sized buckets. For each bucket, we take 25 random typeswithout repetition and select 5 intent profiles from each type. Theresulting sample has a total of 2,010 quadruples (around 1.29% ofthe size of the KB). For the following experiment, we ignore theconfidence scores, so as not to bias annotators. Thus, we shall referto triples, not quadruples from now on.We conduct an annotation experiment to decide whether a tripleis correct w.r.t. its intent profile. Three expert annotators eachmanually labeled the sampled triples. Note that for every profile,the searchedForType triple is trivially correct. Also, if there isonly a single expressedBy triple in the profile, then it is trivially [0, 0.87) [0.87, 0.88) [0.88, 0.9) [0.9, 0.93) [0.93, 1]Confidence intervals according to the splitting percentiles0%20%40%60%80%100% P r opo r t i ono ft r i p l e s OF C ATEGORY
Incorrect,
EXPRESSED B Y Figure 2: Proportion of triples in the annotated sample (y-axis), and number of intent profiles in the KB (on top of eachbar), per confidence bucket. correct. We measure the inter-annotator agreement using FleissKappa coefficient κ , separately on ofCategory and expressedBytriples. For the set of ofCategory triples, κ = . κ = . We have addressed the problem of constructing a knowledge baseof entity-oriented search intents and proposed a pipeline approach.We have performed an experimental evaluation on the level ofindividual components as well as on the end-to-end task. Using asample of 4.5K labeled instances for 50 types as training data, wehave generated a knowledge base of over 30K intents for almost600 unseen types. In future work, we aim to perform extrinsicevaluation, by utilizing our IntentsKB as part of a larger task.
REFERENCES [1] Roi Blanco, Hideo Joho, Adam Jatowt, and Haitao Yu. 2017. Test Collection forEvaluating Actionable Knowledge Graphs. In
Proc. of KG4IR at SIGIR . 32–37.[2] Andrei Broder. 2002. A Taxonomy of Web Search.
SIGIR Forum
36, 2 (2002), 3–10.[3] Darío Garigliotti and Krisztian Balog. 2018. Towards an Understanding of Entity-Oriented Search Intents. In
Proc. of ECIR . 644–650.[4] Thomas Lin, Patrick Pantel, Michael Gamon, Anitha Kannan, and Ariel Fuxman.2012. Active Objects: Actions for Entity-centric Search. In
Proc. of WWW . 589–598.[5] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.Distributed Representations of Words and Phrases and Their Compositionality. In
Proc. of NIPS . 3111–3119.[6] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic Regularitiesin Continuous Space Word Representations. In
Proc. of HLT-NAACL .[7] Ndapandula Nakashole, Tomasz Tylenda, and Gerhard Weikum. 2013. Fine-grainedSemantic Typing of Emerging Entities. In
Proc. of ACL . 1488–1497.[8] Jeffrey Pound, Peter Mika, and Hugo Zaragoza. 2010. Ad-hoc object retrieval inthe web of data. In
Proc. of WWW . 771–780.[9] Ridho Reinanda, Edgar Meij, and Maarten de Rijke. 2015. Mining, Ranking andRecommending Entity Aspects. In