Using Rank Aggregation for Expert Search in Academic Digital Libraries
aa r X i v : . [ c s . I R ] J a n Using Rank Aggregation for Expert Search inAcademic Digital Libraries
Catarina Moreira [email protected]
Bruno Martins [email protected]
P´avel Calado [email protected]
Instituto Superior T´ecnico, INESC-IDAv. Professor Cavaco Silva, 2744-016 Porto Salvo, Portugal
Published in Proceedings of the 3rd Simp´osio de Inform´atica, 2011, Portugal
Abstract
The task of expert finding has been getting increasing attention in informationretrieval literature. However, the current state-of-the-art is still lacking in princi-pled approaches for combining different sources of evidence. This paper exploresthe usage of unsupervised rank aggregation methods as a principled approach forcombining multiple estimators of expertise, derived from the textual contents, fromthe graph-structure of the citation patterns for the community of experts, andfrom profile information about the experts. We specifically experimented two un-supervised rank aggregation approaches well known in the information retrievalliterature, namely CombSUM and CombMNZ. Experiments made over a dataset ofacademic publications for the area of Computer Science attest for the adequacy ofthese methods.
The automatic search for knowledgeable people in the scope of specific user communities,with basis on documents describing people’s activities, is an information retrieval problemthat has been receiving increasing attention [20]. Usually referred to as expert finding ,the task involves taking a short user query as input, denoting a topic of expertise, and
This work was partially supported by the ICP Competitiveness and Innovation Framework Programof the European Commission, through the European Digital Mathematics Library (EuDML) project –
Serdyukov and Macdonald have surveyed the most important concepts and representa-tive previous works in the expert finding task [20, 18]. Two of the most popular andwell-performing types of methods are the profile-centric and the document-centric ap-proaches [8, 24]. Profile-centric approaches build an expert profile as a pseudo document,by aggregating text segments relevant to the expert [2]. These profiles are latter indexedand used to support the search for experts on a topic. Document-centric approachesare typically based on traditional document retrieval techniques, using the documentsdirectly. In a probabilistic approach to the problem, the first step is to estimate theconditional probability p ( q | d ) of the query topic q given a document d . Assuming thatthe terms co-occurring with an expert can be used to describe him, p ( q | d ) can be used toweight the co-occurrence evidence of experts with q in documents. The conditional proba-bility p ( c | q ) of an expert candidate c given a query q can then be estimated by aggregatingall the evidences in all the documents where c and q co-occur. Experimental results showthat document-centric approaches usually outperform profile-centric approaches [24].Many different authors have proposed sophisticated probabilistic retrieval models,specific to the expert finding task, with basis on the document-centric approach [2, 19,20]. For instance Cao et al. proposed a two-stage language model combining documentrelevance and co-occurrence between experts and query terms [6]. Fang and Zhai derived agenerative probabilistic model from the probabilistic ranking principle and extend it withquery expansion and non-uniform candidate priors [12]. Zhu et al. proposed a multiplewindow based approach for integrating multiple levels of associations between experts2nd query topics in expert finding [25]. More recently, Zhu et al. proposed a unifiedlanguage model integrating many document features for expert finding [26]. Althoughthe above models are capable of employing different types of associations among queryterms, documents and experts, they mostly ignore other important sources of evidence,such as the importance of individual documents, or the co-citation patterns betweenexperts available from citation graphs. In this paper, we offer a principled approach forcombining a much larger set of expertise estimates.In the Scientometrics community, the evaluation of the scientific output of a scientisthas also attracted significant interest due to the importance of obtaining unbiased andfair criteria. Most of the existing methods are based on metrics such as the total numberof authored papers or the total number of citations. A comprehensive description ofmany of these metrics can be found in [22, 23]. Simple and elegant indexes, such as theHirsch index, calculate how broad the research work of a scientist is, accounting for bothproductivity and impact. Graph centrality metrics inspired on PageRank, calculated overcitation or co-authorship graphs, have also been extensively used [17].Previous studies have addressed the problem of combining multiple information re-trieval mechanisms through unsupervised rank aggregation, often with basis on methodsthat take their inspiration on voting protocols proposed in the area of statistics and inthe social sciences. Given M voters (i.e., the different estimators of expertise) and N objects (i.e., the experts), we can see each voter as returning an ordered list of the N objects according to their own preferences. From these M ordered lists, the problem ofunsupervised rank aggregation concerns with finding a single consensus list which opti-mally combines the M rankings. There are different methods for addressing the problemwhich, according to Julien Ah-Pine [1], can be divided into two large families of methods: • Positional methods - For each object, we consider the preferences (i.e., the scores)given by each voter, aggregating them through some particular technique and finallyre-ranking objects using the aggregated preferences. The first positional methodwas proposed by Borda, but linear and non-linear combinations of preferences, suchas their arithmetic mean or the triangular norm, are also frequently used [14, 1]. • Majoritarian methods - Pairwise comparison matrices are computed for theobjects, mostly based upon the aggregation of order relations using associationcriteria such as Condorcet’s criterion, or distance criteria such as Kendall’s distance.Other majoritarian methods have also recently been proposed, using Markov chainmodels [10] or techniques from multicriteria decision theory [13].Fox and Shaw [14, 1] defined several rank aggregation techniques (e.g., CombSUM andCombMNZ) which have been the object of much IR research since, including in the areaof expert search [18]. In our experiments, we compared the CombSUM and CombMNZunsupervised rank aggregation methods, which are detailed in Section 3.
Given a set of queries Q = { q , . . . , q | Q | } and a collection of candidate experts E = { e , . . . , e | E | } , each associated with specific documents describing his topics of expertise,3 testing corpus consists of a set of query-expert pairs, each ( q i , e j ) ∈ Q × E , upon which arelevance judgment indicating the match between q i and e j is assigned by a labeler. Thisrelevance judgment can be a binary label, e.g., relevant or non-relevant, or an ordinalrating indicating relevance, e.g., definitely relevant, possibly relevant, or non-relevant.For each instance ( q i , e j ), a feature extractor produces a vector of features that describesthe match between q i and e j . Features can range from classical IR estimators computedfrom the documents associated with the experts (e.g., term frequency, inverse documentfrequency, BM25, etc.) to link-based features computed from networks encoding rela-tions between the experts in E (e.g., PageRank). The inputs of an unsupervised rankaggregation algorithm comprise a set of query-expert pairs corpus, their correspondingfeature vectors, and the corresponding relevance judgments. The output produces a rank-ing score resulting from the aggregation of the multiple features. The relevance of eachexpert e j towards the query q is determined through this aggregated score. In this paper,we experimented with the CombSUM and CombMNZ approaches.The CombSUM and CombMNZ unsupervised rank aggregation algorithms were orig-inally proposed by Fox and Shaw [14]. These algorithms are used to aggregate the infor-mation gathered from different sources (i.e., different features) in order to achieve moreaccurate ranking results than using individual scores. Both CombSUM and CombMNZuse normalized sums for the different features. To perform the normalization, we appliedthe Min-Max Normalization procedure, which is given by Equation 1. N ormalizedV alue = V alue − minV aluemaxV alue − minV alue (1)The CombSUM score of an expert e for a given query Q is the sum of the normalizedscores received by the expert in each individual ranking, and is given by Equation 2. CombSU M ( e, Q ) = k X j =1 score j ( e, Q ) (2)Similarly, the CombMNZ score of an expert e for a given query Q is defined byEquation 3, where r e is the number of non-zero similarities. CombM N Z ( e, Q ) = CombSU M ( e, Q ) × r e (3) The considered set of features for estimating the expertize of a given researcher towards agiven query can be divided into three groups, namely textual features, profile features andnetwork features. The textual features are similar to those used in standard text retrievalsystems (e.g., TF-IDF and BM25 scores). The profile similarity features correspondto importance estimates for the authors, derived from their profile information (e.g.,number of papers published). Finally, the network features correspond to importanceand relevance estimates computed from the author co-authorship and co-citation graphs.4 .1 Textual Similarity Features
To build some of our estimators of expertise, we used the textual similarity betweenthe query and the contents of the documents associated to the candidate experts. In thedomain of academic digital libraries, the associations between documents and experts caneasily be obtained from the authorship information. For each topic-expert pair, we usedthe
OkapiBM
25 document-scoring function, to compute the textual similarity features.Okapi BM25 is a state-of-the-art IR ranking mechanism composed of several simplerscoring functions with different parameters and components (e.g., term frequency andinverse document frequency). It can be computed through the formula in Equation 4,where
Terms ( q ) represents the set of terms from query q , Freq ( i , d ) is the number ofoccurrences of term i in document d , | d | is the number of terms in document d , and A isthe average length of the documents in the collection. The values given to the parameters k and b were 1.2 and 0.75 respectively. Most previous IR experiments use these defaultvalues for the k and b parameters. BM q, d ) = X i ∈ T erms ( q ) log (cid:18) N − F req ( i ) + 0 . F req ( i ) + 0 . (cid:19) × ( k + 1) × F req ( i,d ) | d | F req ( i,d ) | d | + k × (1 − b + b × | d |A ) (4)We also experimented with other textual features commonly used in ad-hoc IR systems,such as Term Frequency (TF) and
Inverse Document Frequency (IDF).Term Frequency (TF) corresponds to the number of times that each individual termin the query occurs in all the documents associated with the author. Equation 5 describesthe TF formula, where i ∈ T erms ( q ) represents the set of terms from query q , j ∈ Docs ( a )is the set of documents having a as author, F req ( i, d j ) is the number of occurrences ofterm i in document d j and | d j | represents the number of terms in document d j . T F q,a = X j ∈ Docs ( a ) X i ∈ T erms ( q ) F req ( i, d j ) | d j | (5)The Inverse Document Frequency (IDF) corresponds to the sum of the values forthe inverse document frequency of each query term and is given by Equation 6. In thisformula, | D | is the size of the document collection and f i,D corresponds to the number ofdocuments in the collection where the i th query term occurs. IDF q = X i ∈ T erms ( q ) log | D | f i,D (6)We also used other simpler features such as the number of unique authors associatedwith documents containing the query topics, the range of years since the first and lastpublications of the author containing the query terms and the document length.In the computation of the textual features, we considered two different sources ofevidence extracted from the documents, namely (i) a stream consisting of the titles, and(ii) a stream using the abstracts of the articles. Separate features were computed for eachof these streams. 5 .2 Profile Information Features We also considered a set of profile features related to the amount of published materialsassociated with authors, generally taking the assumption that highly prolific authors aremore likely to be considered experts. Most of the features based on profile informationare query independent, meaning that they have the same value for different queries. Theconsidered set of profile features are based on the number of publications in conferencesand in journals with and without the query topics in their contents, the average numberof papers and articles per year, and the temporal interval between the first and the lastpublications.
Scientific impact metrics computed over scholarly networks, encoding co-citation and co-authorship information, can offer effective approaches for estimating the importance ofthe contributions of particular publications. Thus, we considered a set of features thatestimate expertise with basis on co-citation and co-authorship information. The consid-ered features are divided in two sets, namely (i) citation counts and (ii) academic indexes.Regarding citation counts, we used the total, the average and the maximum number ofcitations of papers containing the query topics, the average number of citations per yearof the papers associated with an author and the total number of unique collaboratorswhich worked with an author.Regarding academic impact indexes, we used the following features: • Hirsch index of the author and of the author’s institution, measuring both thescientific productivity and the scientific impact of the author or the institution [15].A given author or institution has an Hirsch index of h if h of his N p papers have atleast h citations each, and the other ( N p − h ) papers have at most h citations each.Authors with a high Hirsch index, or authors associated with institutions with ahigh Hirsch index, are more likely to be considered experts. • The h - b -index , which extends the Hirsch index for evaluating the impact of scien-tific topics in general [3]. In our case, the scientific topic is given by the query termsand thus the query has an h - b -index of i if i of the N p papers containing the queryterms in the title or abstract have at least i citations each, and the other ( N p − i )papers have at most i citations each. • Contemporary Hirsch index of the author, which adds an age-related weightingto each cited article, giving less weight to older articles [21]. A researcher has acontemporary Hirsch index h c if h c of his N p articles have a score of S c ( i ) > = h c each, and the rest ( N p − h c ) articles have a score of S c ( i ) < = h c . For an article i ,the score S c ( i ) is defined as: S c ( i ) = γ ∗ ( Y ( now ) − Y ( i ) + 1) − δ ∗ | CitationsT o ( i ) | (7)In the formula, Y ( i ) refers to the year of publication for article i . The γ and δ parameters are set to 4 and 1, respectively, meaning that the citations for an articlepublished during the current year account four times, the citations for an article6ublished 4 years ago account only one time, the citations for an article published6 years ago account 4 / • Trend Hirsch index [21] for the author, which assigns to each citation an expo-nentially decaying weight according to the age of the citation, this way estimatingthe impact of a researcher’s work in a particular time instance. A researcher has atrend Hirsch index h t if h t of his N p articles get a score of S t ( i ) > = h t each, andthe rest ( N p − h t ) articles get a score of S t ( i ) < = h t . For an article i , the score S t ( i ) is defined as shown bellow: S t ( i ) = γ ∗ X ∀ x ∈ C ( i ) ( Y ( now ) − Y ( x ) + 1) − δ (8)Similarly to the case of the contemporary Hirsch index, the γ and δ parameters arehere also set to 4 and 1, respectively. • Individual Hirsch index of the author, computed by dividing the value of thestandard Hirsch index by the average number of authors in the articles that con-tribute to the Hirsch index of the author, in order to reduce the effects of frequentco-authorship with influential authors [4]. • The a -index of the author or the author’s institution, measuring the magnitude ofthe most influential articles. For an author or an institution with an Hirsch indexof h that has a total of N c,tot citations toward his papers, we say that he has an a -index of a = N c,tot /h . • The g -index of the author or his institution, also quantifying scientific productivitywith basis on the publication record [11]. Given a set of articles associated withan author or an institution, ranked in decreasing order of the number of citationsthat they received, the g-index is the (unique) largest number such that the top g articles received on average at least g citations. • The e -index of the author [28] which represents the excess amount of citationsof an author. The motivation behind this index is that we can complement the h -index by taking into account these excess amounts of citations which are ignoredby the h -index. The e -index is given by the Equation 9: e = h X j =1 p cit j − h (9)In the above equation, cit j are the citations received by the j t h paper and h is the h -index.We also followed the ideas of Chen et al. [7] by considering a set of network featuresthat estimate the influence of individual authors using PageRank, a well-known graphlinkage analysis algorithm that was introduced by the Google search engine [5]. PageRankassigns a numerical weighting to each element of a linked set of objects (e.g., hyperlinkedWeb documents or articles in a citation network) with the purpose of measuring its7elative importance within the set. The PageRank value of a node is defined recursivelyand depends on the number and PageRank scores of all other nodes that link to it (i.e.,the incoming links). A node that is linked to by many nodes with high PageRank receivesa high rank itself.Formally, given a graph with N nodes i = 1 , , · · · , N , with L directed links thatrepresent references from an initial node to a target node with weights α = 1 , , · · · , L ,the PageRank P r i for the i th node is defined by: P r i = 0 . N + 0 . X j ∈ inlinks ( L,i ) α j P r j outlinks ( L, j ) (10)In the formula, the sum is over the neighboring nodes j in which a link points to node i . The first term represents the random jump in the graph, giving a uniform injectionof probability into all nodes in the graph. The second term describes the propagation ofprobability corresponding to a random walk, in which a value at node j propagates tonode i with probability α j P r j outlinks ( L,j ) .The PageRank-based features that we considered correspond to the sum and average ofthe PageRank values associated to the papers of the author that contain the query terms,computed over a directed graph representing citations between papers. Each citation linkin the graph is given a score of 1 /N , where N represents the number of authors in thepaper. Authors with high PageRank scores are more likely to be considered experts. The main hypothesis behind this work is that unsupervised rank aggregation approachescan be effectively used in the context of expert search tasks, in order to combine differentestimators of relevance in a principled way, this way improving over the current state-of-the art. To validate this hypothesis, we have built a prototype expert search system, usingtwo unsupervised rank aggregation methods, namely the CombSUM and CombMNZmethods.We implemented the methods responsible for computing the features listed in theprevious section, using the
Microsoft SQL Server 2008 relational database (e.g., thefull-text search capabilities for computing the textual similarity features) together withexisting Java software packages (e.g., the LAW package for computing PageRank).The validation of the prototype required a sufficiently large repository of textualcontents describing the expertise of individuals within a specific area. In this work, weused a dataset for evaluating expert search in the Computer Science research domain,corresponding to an enriched version of the DBLP database made available through theArnetminer project. DBLP data has been used in several previous experiments regardingcitation analysis [22, 23] and expert search [9]. It is a large dataset covering both journaland conference publications, and where substantial effort has been put into resolvingthe problem of author identity resolution, i.e., references to the same persons with othernames. http://law.dsi.unimi.it/software.php Dataset Property ValueTotal Authors 1 033 050Total Publications 1 632 440Total Publications containing Abstract 653 514Total Papers Published in Conferences 606 953Total Papers Published in Journals 436 065Total Number of Citations Links 2 327 450
Table 1: Statistical characterization for the DBLP dataset used in our experiments.To validate the different learning to rank methods, we also needed a set of queries withthe corresponding author relevance judgments. We used the relevant judgments providedby Arnetminer which have already been used in other expert finding experiments [27].The Arnetminer dataset comprises a set of 13 query topics, each associated to a list ofexpert authors.In order to add negative relevance judgments (i.e., complement the dataset with unim-portant authors for each of the query topics), we searched the dataset with the keywordsassociated to each topic, retrieving the top n/ n/ n corresponds tothe number of expert authors associated to each particular topic. Table 2 shows thedistribution for the number of experts associated to each topic in the collection. Query Topics Authors Query Topics Authors
Boosting (B) 46 Natural Language (NL) 41Computer Vision (CV) 176 Neural Networks (NN) 103Cryptography (C) 148 Ontology (O) 47Data Mining (DM) 318 Planning (P) 23Information Extraction (IE) 20 Semantic Web (SW) 326Intelligent Agents (IA) 30 Support Vector Machines (SVM) 85Machine Learning (ML) 34
Table 2: Characterization of the Arnetminer dataset of Computer Science experts.To measure the quality of the results produced by the different rank aggregationalgorithms, we used two different performance metrics, namely the Precision at k (P@k)and the Mean Average Precision (MAP).Precision at rank k is used when a user wishes only to look at the first k retrieveddomain experts. The precision is calculated at that rank position through Equation 11. P @ k = r ( k ) k (11) http://arnetminer.org/lab-datasets/expertfinding/
9n the formula, r ( k ) is the number of relevant authors retrieved in the top k positions. P @ k only considers the top-ranking experts as relevant and computes the fraction of suchexperts in the top- k elements of the ranked list.The Mean of the Average Precision over test queries is defined as the mean over theprecision scores for all retrieved relevant experts. It is given by: M AP [ r ] := P nk =1 P @ k [ r ] × I { g r k = max( g ) } P nk =1 I { g r k = max( g ) } (12)As before, n is the number of experts associated with query q . In the case of ourdatasets, max( g ) = 1 (i.e., we have 2 different grades for relevance, 0 or 1).Table 3 presents the obtained results over the dataset, when considering the completeset of features described in Section 4. The obtained results attest for the adequacy ofboth unsupervised rank aggregation approaches, showing that CombSUM and CombMNZachieve a similar performance, with CombMNZ slightly outperforming CombSUM, interms of MAP. In a separate experiment, we attempted to measure the impact of the P@5 P@10 P@15 P@20 MAPCombSUM 0.5076 0.4846 0.4769 0.5115 0.5266CombMNZ
Table 3: Results of the CombSUM and CombMNZ methods.different types of ranking features on the quality of the results. Using the best performingrank aggregation algorithm, namely the CombMNZ method, we separately measuredthe results obtained by using approaches that considered (i) only the textual similarityfeatures, (ii) only the profile features, (iii) only the network features, (iv) textual similarityand profile features, (v) textual similarity and network features and (vi) profile andnetwork features. Table 4 shows the obtained results, where we also compare them withthe previous results reported by Yang et al. [27] for their supervised approach for expertfinding.
P@5 P@10 P@15 P@20 MAPText Similarity + Profile + Network 0.6000 0.6077 0.6141 0.6256 0.5832Text Similarity + Profile 0.5231 0.5615 0.5487 0.5577 0.5469Text Similarity + Network 0.5538 0.5692 0.5782 0.5718 0.5655Profile + Network 0.6923 0.6308 0.6205 0.6077 0.5986Text Similarity 0.5231 0.5154 0.5436 0.5231 0.5538Profile 0.5846 0.5769 0.5897 0.5923 0.5895Network 0.6462 0.6462 0.6121 0.6128 0.5990Expert Finding (Yang et al.) [27] 0.5500 0.6000 0.6333 – 0.6356
Table 4: The results obtained with the different sets of features.Since DBLP has rich information about citation links, we can see that the set ofnetwork features achieve the best results for this dataset in terms of MAP. The resultsalso show that, individually, textual similarity features have the poorest results. Thismeans that considering only textual evidence provided by query topics, together witharticle’s titles and abstracts, may not be enough to determine if some authors are expertsor not, and that indeed the information provided by citation and co-authorship patternscan help in expert retrieval. Finally, when comparing our unsupervised method against10he supervised learning to rank approach proposed by Yang et al. [27], showing thatour approach provides very competitive results against the supervised method. Noticethat unsupervised approaches are particularly interesting in the context of expert searchsystems for academic digital libraries, since relevance judgments for specific areas ofknowledge, which are required to the usage of supervised approaches, are hard to obtain.
This paper argued that unsupervised rank aggregation methods provide a sound approachfor combining multiple estimators of expertise, derived from the textual contents, fromthe graph-structure of the community of experts, and from expert profile information.Experiments on a dataset of academic publications show very competitive results interms of P@5 and MAP, attesting for the adequacy of the proposed approaches. This isparticularly interesting to the application domain of academic expert search, since therelevance judgments required by supervised approaches are only scarcely available.Despite the interesting results, there are also many ideas for future work. Recentworks have, for instance, proposed that there are advanced unsupervised rank aggregationmethods capable of outperforming CombSUM and CombMNZ. This is currently a veryhot topic of research and, for future work, we would for instance like to experiment withthe ULARA algorithm recently proposed by Klementiev et al. [16].
References [1] Julien Ah-Pine. On data fusion in information retrieval using different aggregationoperators.
Web Intelligence and Agent Systems , 9(1), 2011.[2] K. Balog, L. Azzopardi, and M. de Rijke. Formal models for expert finding inenterprise corpora. In
Proceedings of the 29th annual international ACM SIGIRconference on Research and development in information retrieval , 2006.[3] M. Banks. An extension of the Hirsch index: Indexing scientific topics and com-pounds.
Scientometrics , 69(1), 2006.[4] P. D. Batista, M. G. Campiteli, O. Kinouchi, and A. S. Martinez. Is it possible tocompare researchers with different scientific interests?
Scientometrics , 68(1), 2006.[5] S. Brin, L. Page, R. Motwani, and T. Winograd. The pagerank citation ranking:Bringing order to the web. Technical Report 1999-66, Stanford Digital LibraryTechnologies Project, 1999.[6] Y. Cao, J. Liu, S. Bao, and H. Li. Research on expert search at enterprise track ofTREC 2005. In
Proceedings of the 14th Text REtrieval Conference , 2006.[7] P. Chen, H. Xie, S. Maslov, and S. Redner. Finding scientific gems with Google’spage rank algorithm.
Journal of Informetrics , 1(1), 2007.[8] N. Craswell, A. P. de Vries, and I. Soboroff. Overview of the TREC-2005 enterprisetrack. In
Proceedings of the 14th Text REtrieval Conference , 2006.119] H. Deng, I. King, and M. R. Lyu. Formal models for expert finding on DBLPbibliography data. In
Proceedings of the 8th IEEE International Conference onData Mining , 2008.[10] Cynthia Dwork, Ravi Kumar, Moni Naor, and D. Sivakumar. Rank aggregationmethods for the web. In
Proceedings of the 10th international conference on WorldWide Web , 2001.[11] L. Egghe. Theory and practise of the g -index. Scientometrics , 69(1), 2006.[12] H. Fang and C. Zhai. Probabilistic models for expert finding. In
Proceedings of the29th European Conference on Information Retrieval Research , 2007.[13] Mohamed Farah and Daniel Vanderpooten. An outranking approach for rank aggre-gation in information retrieval. In
Proceedings of the 30th annual international ACMSIGIR conference on Research and development in information retrieval , 2007.[14] E. A. Fox and J. A. Shaw. Combination of multiple searches. In
Proceedings of the2nd Text Retrieval Conference , 1994.[15] J. E. Hirsch. An index to quantify an individual’s scientific research output.
Pro-ceedings of the National Academy of Sciences USA , 102(46), 2005.[16] Alexandre Klementiev, Dan Roth, Kevin Small, and Ivan Titov. Unsupervised rankaggregation with domain-specific expertise. In
Proceedings of the 21st InternationalJoint Conference on Artifical intelligence , 2009.[17] X. Liu, J. Bollen, M. L. Nelson, and H. Van de Sompel. Co-authorship networks inthe digital library research community.
Information Processing and Management ,41(6), 2005.[18] C. Macdonald and I. Ounis. Voting techniques for expert search.
Knowledge andInformation Systems , 16(3), 2008.[19] D. Petkova and W. B. Croft. Proximity-based document representation for namedentity retrieval. In
Proceedings of the 16th ACM Conference on Information andKnowledge Management , 2007.[20] P. Serdyukov.
Search for Expertise : Going Beyond Direct Evidence . PhD thesis,University of Twente, 2009.[21] A. Sidiropoulos, D. Katsaros, and Y. Manolopoulos. Generalized h -index for disclos-ing latent facts in citation networks. Scientometrics , 2006.[22] A. Sidiropoulos and Y. Manolopoulos. A citation-based system to assist prize award-ing.
ACM SIGMOD Record , 34(4), 2005.[23] A. Sidiropoulos and Y. Manolopoulos. Generalized comparison of graph-based rank-ing algorithms for publications and authors.
Journal for Systems and Software ,79(12), 2006. 1224] I. Soboroff, A. P. de Vries, and N. Craswell. Overview of the TREC-2006 enterprisetrack. In
Proceedings of the 15th Text REtrieval Conference , 2007.[25] J. Zhu S. Song, S. R¨uger, M. Eisenstadt, and E. Motta. The open university atTREC 2006 enterprise track expert search task. In
Proceedings of the 15th TextREtrieval Conference , 2007.[26] J. Zhu S. Song, S. R¨uger, and J. Huang. Modeling document features for expertfinding. In
Proceedings of the 17th ACM Conference on Information and KnowledgeManagement , 2008.[27] Y. Yang, J. Tang, B. Wang, J. Guo, J. Li, and S. Chen. Expert2Bole: From expertfinding to bole search. In
Knowledge Discovery and Data Mining , 2009.[28] Chun-Ting Zhang. The e-index, complementing the h-index for excess citations.