A Scalable Hybrid Research Paper Recommender System for Microsoft Academic
AA Scalable Hybrid Research Paper Recommender System forMicrosoft Academic
Anshul Kanakia
Microsoft ResearchRedmond, Washington [email protected]
Zhihong Shen
Microsoft ResearchRedmond, Washington [email protected]
Darrin Eide
Microsoft ResearchRedmond, Washington [email protected]
Kuansan Wang
Microsoft ResearchRedmond, Washington [email protected]
ABSTRACT
We present the design and methodology for the large scale hy-brid paper recommender system used by Microsoft Academic. Thesystem provides recommendations for approximately 160 millionEnglish research papers and patents. Our approach handles in-complete citation information while also alleviating the cold-startproblem that often affects other recommender systems. We use theMicrosoft Academic Graph (MAG), titles, and available abstracts ofresearch papers to build a recommendation list for all documents,thereby combining co-citation and content based approaches. Tun-ing system parameters also allows for blending and prioritizationof each approach which, in turn, allows us to balance paper noveltyversus authority in recommendation results. We evaluate the gen-erated recommendations via a user study of 40 participants, withover 2400 recommendation pairs graded and discuss the qualityof the results using P@10 and nDCG scores. We see that there isa strong correlation between participant scores and the similarityrankings produced by our system but that additional focus needs tobe put towards improving recommender precision, particularly forcontent based recommendations. The results of the user survey andassociated analysis scripts are made available via GitHub and therecommendations produced by our system are available as part ofthe MAG on Azure to facilitate further research and light up novelresearch paper recommendation applications.
KEYWORDS recommender system, word embedding, big data, k-means, cluster-ing, document collection
Microsoft Academic (MA) is a semantic search engine for academicentities [26]. The top level entities of the Microsoft Academic Graph(MAG) include papers and patents, fields of study, authors, affilia-tions (institutions and organizations), and venues (conferences andjournals), as seen in Fig. 1. As of October, 2018, there are over 200million total documents in the MA corpus, of which approximately160 million are English papers or patents. These figures are growingrapidly [11].The focus of this article is to present the recommender systemand paper similarity computation platform for English research https://preview.academic.microsoft.com papers developed for the MAG. We henceforth define the term‘paper’ to mean English papers and patents in the MAG, unlessotherwise noted. Research paper recommendation is a relativelynascent field [3] in the broader recommender system domain butits value to the research community cannot be overstated.The current approach to knowledge discovery is largely manual— either following the citation graph of known papers or throughhuman curated approaches. Additionally, live feeds such as pub-lisher RSS feeds or manually defined news triggers are used to gainexposure to new research content. This often leads to incompleteliterature exploration, particularly by novice researchers since theyare almost completely reliant on what they see online or their advi-sors and peer networks for finding new relevant papers. Followingthe citation network of known papers to discover content oftenleads to information overload, resulting in dead ends and hoursof wasted effort. The current manual approaches are not scalable,with tens of thousands of papers being published everyday andthe number of papers published globally increasing exponentially[11, 12]. To help alleviate this problem, a number of academicsearch engines have started adding recommender systems in recentyears [1, 7, 18, 19, 28]. Still other research groups and independentcompanies are actively producing tools to assist with research pa-per recommendations for both knowledge discovery and citationassistance [2, 5, 13, 17]; both these use cases being more-or-lessanalogous.Existing paper recommender systems suffer from a number oflimitations. Besides Google Scholar, Microsoft Academic, Seman-tic Scholar, Web of Science, and a handful of other players, thevast majority of paper search engines are restricted to particularresearch domains such as the PubMed database for Medicine &Biology, and IEEE Xplore for Engineering disciplines. As such, it isimpossible for recommender systems on these field-specific searchsites to suggest cross-domain recommendations. Also, a numberof proposed user recommender systems employ collaborative fil-tering for generating user recommendations [9]. These systemssuffer from the well known cold-start problem . There is little to noreadily available data on metrics such as user attachment rates forresearch paper search and recommendation sites and so it becomesdifficult to evaluate the efficacy of collaborative filtering techniques Some collaborative filtering approaches assume paper authors will be system usersand use citation information to indicate user intent. Even so, new authors/users stillsuffer from cold-start problem a r X i v : . [ c s . D L ] M a y igure 1: The Microsoft Academic Graph (MAG) website preview and statistics as of October, 2018. without a solid active user base. Moreover, with the introduction ofprivacy legislation such as the General Data Protection Regulationact (GDPR) in Europe, it is becoming increasing difficult and costlyto rely on user data — which is why the Microsoft Academic websitedoes not store personal browsing information — making collabora-tive filtering all the more difficult. Finally, besides Google Scholar,the self attested paper counts of other research paper databasesare in the tens of millions while the estimated number of papersfrom our system (as well as Google Scholar estimates) put the to-tal number of published papers easily in the hundreds of millions.Having a tenth of the available research corpus can heavily diluterecommendations, providing incomplete and unsatisfactory results.The MA paper recommender platform aims to alleviate some of theaforementioned shortcomings by,(1) employing the entire MAG citation network and interdis-ciplinary corpus of over 200 million papers,(2) using a combination of co-citation based and content em-bedding based approaches to maximize recommendationcoverage over the entire corpus while circumventing thecold-start problem,(3) and providing the computed paper recommendations tothe broader research and engineering community so theycan be analyzed and improved by other research groups.We present two possible interaction modes with the MA paperrecommender platform. The first mode is via the “Related Papers”section on the MA search engine paper details page. Users canbrowse the MA search site using the novel semantic interpretationengine powering the search experience to view their desired paper[26]. The paper details page, as seen in Fig.2, contains a tab forbrowsing related papers (with dynamic filters) that is populated us-ing the techniques mentioned here. The second mode of interactionis via Microsoft Azure Data Lake (ADL) services. The entire MAGis published under the ODC-By open data license and availablefor use via Microsoft Azure. Users can use scripting languagessuch as U-SQL and python not just to access the pre-computedpaper recommendations available as part of the MAG but also togenerate on-the-fly paper recommendations for arbitrary text input https://opendatacommons.org/licenses/by/1.0/ in a performant manner. The means by which this functionality isachieved is described in the following sections. Figure 2: The “Related Papers” tab on Microsoft Academicwebsite paper details page with additional filters visible onthe left pane.
The complexity of developing a recommendation system for MAstems from the following feature requirements: • Coverage : Maximize recommendation coverage over theMAG corpus. • Scalability : Recommendation generation needs to be donewith computation time and storage requirements in mindas the MAG ingests tens of thousands of new papers eachweek. • Freshness : All new papers regularly ingested by the MAdata pipeline must be assigned related papers and may hemselves be presented as ‘related’ to papers already inthe corpus. • User Satisfaction : There needs to be a balance betweenauthoritative recommendations versus novel recommenda-tions so that newer papers are discoverable without com-promising the quality of recommendations.To tackle these requirements we have developed a hybrid recom-mender system that uses a tunable mapping function to combineboth content-based (CB) and co-citation based (CcB) recommen-dations to produce a final static list of recommendations for everypaper in the MAG. As described in [8], our approach employs a weighted mixed hybridization approach. Our content-based ap-proach is similar to recent work done on content embedding basedcitation recommendation [5] but differs mainly in the fact that weemploy clustering techniques for additional speedup. Using purelypairwise content embedding similarity for nearest neighbor searchis not viable as this is an O ( n ) problem over the entire paper corpus,which in our case would be over 2 . × similarity computations. The process outlined in this section describes how the paper recom-mendations seen on the MA website, as well as those published inthe MAG Azure database, are generated. More information on theMAG Azure database is available online but the important thingto note is that our entire paper recommendation dataset is openlyavailable as part of this database, if desired.The recommender system uses a hybrid approach of CcB andCB recommendations. The CcB recommendations show a highpositive correlation with user generated scores, as discussed insection 3, and so we consider them to be of high quality. Butthe CcB approach suffers from low coverage since paper citationinformation is often difficult to acquire. To combat this issue wealso use content embedding similarity based recommendations.While CB recommendations can be computationally expensive,and of lower quality than CcB recommendations they have themajor advantages of freshness and coverage. Only paper metadataincluding titles, keywords and available abstracts are needed togenerate these recommendations. Since all papers in the MAG arerequired to have a title, we can generate content embeddings forall English documents in the corpus, just relying on the title if needbe.The resulting recommendation lists of both approaches are finallycombined using a parameterized weighting function (see section2.3) which allows us to directly compare each recommendationpair from either list and join both lists to get a final paper recom-mendation list for nearly every paper in the graph. These lists aredynamically updated as new information comes into the MAG, suchas new papers, paper references or new paper metadata such asabstracts. The recommendation lists are thus kept fresh from weekto week. Consider a corpus of papers P = { p , p , p . . . p n } . We use c i , j = p i is citing p j , 0 otherwise. The co-citation count between https://docs.microsoft.com/en-us/academic-services/graph/ p i and p j is defined as: cc i , j = n (cid:213) k = c i , k c j , k (1)When c i , j ≥
1, we call that p i is a co-citation of p j and vice-versa.Notice, cc i , j = cc j , i . This method presumes that papers with higherco-citation counts are more related to each other. Alternativelyif two papers are often cited together, they are more likely to berelated. This approach of recommendation generation is not new,it was originally proposed in 1973 by Small et al. [27]. Since thenothers such as [14] have built upon the original approach by incorpo-rating novel similarity measures. While we stay true to the originalapproach in this paper, we are investigating other co-citation andnetwork based similarity measures as future work.CcB similarity empirically resembles human behavior when itcomes to searching for similar or relevant material. Having accessto paper reference information is a requirement for generating CcBrecommendations. This presents a challenge since reference listsare often incomplete or just unavailable for many papers, particu-larly for older papers that were published in print but not digitally.Moreover, CcB recommendations are biased towards older paperswith more citations by their very nature. CcB recommendationsprove ineffective for new papers that do not have any citationsyet and therefore cannot have any co-citations. The MAG containscomplete or partial reference information for 31 .
19% of papers, witheach paper averaging approx. 20 references. As a result, only about32 .
5% of papers have at least one co-citation. Nevertheless, CcBrecommendations tend to be of high quality and therefore cannotbe overlooked.
CB recommendations pro-vide a few crucial benefits to overcome the information limitationsfrom CcB recommendations as well as the privacy concerns, systemcomplexity, and cold-start problem inherent in other user basedrecommender systems approaches such as collaborative filtering.CB recommendations only require metadata about a paper that iseasy to find such as its title, keywords, and abstract. We use thistextual data from the MAG to generate word embeddings usingthe well known word2vec library [22, 23]. These word embeddingsare then combined to form paper embeddings which can then bedirectly compared using established similarity metrics like cosinesimilarity [21].Each paper is vectorized into a paper embedding by first trainingword embeddings, w , using the word2vec library. The parametersused in word2vec training are provided in Table. 1 The training datafor word2vec are the titles, and abstracts of all English papers inthe MA corpus. At the same time, we compute the term frequencyvectors for each paper (TF) as well as the inverse document fre-quency (IDF) for each term in the training set. A normalized linearcombination of word vectors weighted by their respective TF-IDFvalues is used to generate a paper embedding, D . Terms in titles andkeywords are weighed twice as much terms in abstracts, as seenin Eqn. 2. This approach has been applied before for CB documentembedding generation, as seen in [24], where the authors assigned × weight to words in paper titles compared to words in the ab-stract. Finally, the document embedding is normalized, ˆ D = D /| D | .Since we use cosine similarity as a measure of document relevance[21] embedding normalization makes the similarity computationstep more efficient since the norm of each paper embedding doesnot need to computed every time it is compared to other papers,just the value of the dot product between embeddings is sufficient. D = . (cid:213) w ∈ titlew ∈ keywords T FIDF ( w ) . ˆ w + (cid:213) w ∈ abstract T FIDF ( w ) . ˆ w (2) Our major contribution tothe approach of CB recommendation is improving scalability usingclustering for very large datasets. The idea to use spherical k-meansclustering for clustering large text datasets is presented in [10]. Theauthors of [10] do a fantastic job of explaining the inherent proper-ties of document clusters formed and make theoretical claims aboutconcept decompositions generated using this approach. Besidesthe aforementioned benefits, our desire to use k-means clusteringstems from the speedup it provides over traditional nearest neigh-bor search in the continuous paper embedding space. We utilizespherical k-means clustering [15] to drastically reduce the numberof pairwise similarity computations between papers when generat-ing recommendation lists. Using trained clusters drops the cost ofpaper recommendation generation from a O ( n ) operation in thetotal number of papers to a O ( n ∗ (| c | + λ c )) operation, where | c | isthe trained cluster count and λ c is the average size of a single papercluster. Other possible techniques commonly used in CB paperrecommender system optimization, such as model classifiers [4],and trained neural networks [5] were investigated but ultimatelyrejected due to the their considerable memory and computationfootprint compared to the simpler k-means clustering approach,particularly for the very large corpus size we are dealing with.Cluster centroids are initialized with the help of the MAG topicstamping algorithm[25]. Papers in the MAG are stamped accordingto a learned topic hierarchy resulting in a directed acyclic graphof topics, such that every topic has at least one parent, with theroot being the parent or ancestor of every single field of study. Asof October, 2018 there are 229 ,
370 topics in the MAG but whenthe centroids for clustering were originally generated — almost ayear ago — there were about 80 ,
000 topics. Of these, 23 ,
533 weretopics with no children in the hierarchy, making them the mostfocused or narrow topics with minimal overlap to other fields. Thetopic stamping algorithm also assigns a confidence value to thetopics stamped for each paper. By using papers stamped with a highconfidence leaf node topic we can guess an initial cluster count aswell as generate initial centroids for these clusters. Therefore, ourinitial centroid count k = , k-means init. clusters ( k ) max iter. ( n ) min error ( δ )23 ,
533 10 10 − word2vec method emb. size loss fn. skipgram ns window size max iter. min-count cutoff10 10 10sample negative10 − Table 1: Parameters used for spherical k-means clusteringand word2vec training. centroids for initializing clusters. The rest of the hyper parametersused for k-means clustering are provided in Table. 1.K-means clustering then progresses as usual until the clustersconverge or we complete a certain number of training epochs. Clus-ter sizes range anywhere from 51 papers to just over 300 ,
000 with93% of all papers in the MAG belonging to clusters of size 35 , O ( n ) to O ( n ∗ (| c | + λ c )) because each paper embedding nowneed only be compared to the embeddings of other papers in thesame cluster. Here | c | = k and λ c is the average cluster size. For asingle paper the complexity of recommendation generation is justthe second term, i.e. | c | + λ c i where c i is the size of the cluster- i that the paper belongs to so if λ c i gets too large then it dominatesthe computation time. In our pipeline we found that the largest 100clusters ranged in size from about 40 ,
000 to 300 ,
000 and took upmore than 40% of the total computation time of the recommenda-tion process. We therefore limit the cluster sizes that we generaterecommendation for to 35 , Finally, both CcB and CB candidate sets for a paper are combined tocreate a unified final set of recommendations for papers in the MAG.CcB candidate sets have co-citation counts associated with eachpaper-recommendation pair. These co-citation counts are mappedto a score between ( , ) to make it possible to directly comparethem with the CB similarity metric. The mapping function used isa modified logistic function as seen in Eq. 3. σ (cid:0) cc i , j (cid:1) = + e θ ( τ − cc i , j ) (3) θ and τ are tunable parameters for controlling the slope and offsetof the logistic sigmoid, respectively. Typically, these values can beestimated using the mean and variance of the domain distributionor input distribution to this function, under the assumption thatthe input distribution is Gaussian-like. While the distribution of co-occurrence counts tended to be more of a Poisson-like distributionwith a long tail, the majority of co-occurrence counts (the mass f the distribution) was sufficiently Gaussian-like. We settled onvalues τ = θ = . [ , ] comparing them becomes trivialand generating a unified recommendation list involves orderingrelevant papers from both lists just based on their similarity to thetarget paper. We evaluated the results of the recommender system via an onlinesurvey. The survey was set up as follows. On each page of the study,participants were presented with a pair of papers. The top paperwas one that had been identified in the MAG as being authored (orco-authored) by the survey participant while the bottom paper wasa recommendation generated using the hybrid recommender plat-form described in the previous section. Metadata for both papers aswell as hyperlinks to the paper details page on the MA website werealso presented to the participant on this page. The participant wasthen asked to judge — on a scale of 1 to 5, with 1 being not relevant to 5 being very relevant — whether they thought the bottom paperwas relevant to the top paper (See Fig. 3). Participants could decideto skip a survey page if they were not comfortable providing a scoreand carry on with the remainder of the survey.The dataset of paper/recommended-paper pairs to show for aparticular participant were generated randomly selecting at most5 of that participant’s authored papers. This was done to ensurefamiliarity with the papers the participants were asked to grade,which we thought would make the survey less time consumingthereby resulting in a higher response rate. Note that while par-ticipants were guaranteed to be authors of the papers, they maynot have been authors of the recommended papers. For each of aparticipant’s 5 papers, we then generated at most 10 recommen-dations using the CcB approach, and 10 recommendations usingthe CB approach. Some newer papers may have had fewer than10 co-citation recommendations. This resulted in each participanthaving to rate at most 100 recommendations. All participants wereactive computer science researchers and so the survey, as a whole,was heavily biased towards rating computer science papers. Wewish to extend this survey to other domains as future work sincethe MAG contains papers and recommendations from tens of thou-sands of different research domains. For now, we limited our scopeto computer science due to familiarity, the ease of participant accessand confidence in participant expertise in this domain.
The user survey was sent to all full-time researchers at MicrosoftResearch and a total of 40 users responded to the survey, result-ing in 2409 scored recommendation pairs collected. Of these, 984were CcB recommendations, 15 recommendation pairs that wereboth content and co-citation, and 1410 CB recommendation pairs. The raw result dataset is available on GitHub . Since at most 10recommendations were presented to a user using each of the twomethods, we computed P@10 for CcB and CB recommendations aswell as P@10 for combined recommendations.Since we did not include any type of explicit score normalizationfor participants during the survey, Table 2 shows precision com-puted assuming, both a user score of at least 3 as a true positiveresult and another row assuming a score of at least 4 as a truepositive result. Recall that users were asked to score recommendedpaper pairs on a scale of 1 being not relevant to 5 being most rele-vant. We also compute the normalized discounted cumulative gain(nDCG) for each of the three methods. Note that we use exponen-tial gain, ( score − )/ loд ( rank + ) instead of linear gain whencomputing DCG. CcB CB CombinedP@10-3 .
315 0 .
226 0 . P@10-4 .
271 0 .
145 0 . nDCG .
851 0 .
789 0 . Table 2: Evaluation metrics for CcB, CB and combined rec-ommender methods. P@10-N indicates that a user score atleast N between [ , ] is considered a true positive. We generated Fig. 4 to see how the similarities computed us-ing the MA recommender system lined up with user scores fromthe study. Each bar in the figure is generated by first aggregatingthe similarity scores for all paper recommendation pairs that weregiven a particular score by users. The first column is all paper rec-ommendation pairs given a score of 1 and so on. Each column isthen divided into sections by binning the paper recommendationpairs according to the similarity computed using the combinedrecommender method, e.g. The orange section of the leftmost bardenotes all paper recommendation pairs with a similarity between [ . , . ) that were given a score of 1 by the user study participants.While the absolute values of the similarity is not very important,what is important to gather from this figure is that it shows a clearpositive correlation between the similarity values computed by thehybrid recommender platform and user scores. This would seemto indicate that recommendation pairs with higher computed simi-larity are more likely to be relevant for users, which is the desiredoutcome for any recommender system. This fact is reinforced bythe nDCG values of each of the recommender methods. A com-bined nDCG of 0 .
891 indicates a strong correlation between thesystem’s computed rankings and those observed from the surveyparticipants.On the other hand, the observed precision values indicate thatthere is room for improving user satisfaction of the presented rec-ommendations, particularly for the CB method. While we expectedCcB recommendations to outperform CB results, having CB recom-mendations be half as precise as CcB recommendations would seemto indicate that additional effort needs to be spent in improving CBrecommender quality. https://github.com/akanakia/microsoft-academic-paper-recommender-user-study igure 3: A screenshot of the online recommender system survey.Figure 4: The distribution of similarity ranges over user re-sponses ranging from being not relevant to being mostrelevant . A natural question to ask about this approach is why not use othervectorization techniques such as doc2vec[20], fasttext[6] or evenincorporate deep learning language models such as ULMFit[16]?While we settled on word2vec for the current production system,we are constantly evaluating other techniques and have this taskset aside as future work. A good experiment would be to gener-ate a family of embeddings and analyze recommendation results,perhaps with the aid of a follow-up user study to understand theimpact of different document vectorization techniques on the re-sult recommendation set. Another avenue for further research liesin tuning the weights and hyper-parameters of the recommender,such as the θ and τ parameters in the co-citation mapping function(Eq. 3). We hypothesize that a reinforcement learning approachcould be used to learn these parameters, given the user study aslabeled ground-truth data for the training model. In the broader scope of evaluating research paper recommendersystems, there is a notable lack of literature that compares existingdeployed technologies. Furthermore, there is a general lack of dataon metrics such as user adoption and satisfaction, and no consensuson which approaches — like the content and co-citation hybrid pre-sented in this paper, or collaborative filtering, and graph analysis toname a few others — prove the most promising in helping to tacklethis problem. Part of the reason for this is that large knowledgegraphs and associated recommender systems are often restrictedbehind paywalls, not open access or open source and hence diffi-cult to analyze and compare. We hope to at least partly alleviatethis problem by providing the entire MAG and precomputed paperrecommendations under the open data license, ODC-By, so thatother researchers may easily use our data and reproduce the resultspresented here as well as conduct their own research and analysison our knowledge graph.In conclusion, we presented a scalable hybrid paper recom-mender platform used by Microsoft Academic that used co-citationand content based recommendations in maximize coverage, scal-ability, freshness, and user satisfaction. We examined the qualityof results produced by the system via a user study and showed astrong correlation between our system’s computed similarities anduser scores for pairs of paper recommendations. Finally, we madethe results of our user study as well the actual recommendation listsused by MA available to researchers to analyze and help furtherresearch in research paper recommender systems. REFERENCES [1] Luiz Barroso. 2006. Exploring the scholarly neighborhood [Blog Post]. https://googleblog.blogspot.com/2006/08/exploring-scholarly-neighborhood.html.[2] Joeran Beel, Akiko Aizawa, Corinna Breitinger, and Bela Gipp. 2017. Mr. DLib:recommendations-as-a-service (RaaS) for academia. In
Digital Libraries (JCDL),2017 ACM/IEEE Joint Conference on . IEEE, 1–2.[3] Joeran Beel, Bela Gipp, Stefan Langer, and Corinna Breitinger. 2016. Research-paper recommender systems: a literature survey.
International Journal on DigitalLibraries
17, 4 (Nov. 2016), 305–338.[4] Steven Bethard and Dan Jurafsky. 2010. Who should I cite: learning literaturesearch models from citation behavior. In
Proceedings of the 19th ACM internationalconference on Information and knowledge management . 609–618.
5] Chandra Bhagavatula, Sergey Feldman, Russell Power, and Waleed Ammar. 2018.Content-Based Citation Recommendation. In
NAACL HLT 2018: 16th AnnualConference of the North American Chapter of the Association for ComputationalLinguistics: Human Language Technologies . 238–251.[6] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. En-riching word vectors with subword information. arXiv preprint arXiv:1607.04606 (2016).[7] Kurt D Bollacker, Steve Lawrence, and C Lee Giles. 1999. A system for automaticpersonalized tracking of scientific literature on the web. In
Proceedings of thefourth ACM conference on Digital libraries . ACM, 105–113.[8] Robin D. Burke. 2002. Hybrid Recommender Systems: Survey and Experiments.
User Modeling and User-adapted Interaction
12, 4 (2002), 331–370.[9] Tsung Teng Chen and Maria R. Lee. 2018. Research Paper Recommender Systemson Big Scholarly Data.. In
PKAW . 251–260.[10] Inderjit S. Dhillon and Dharmendra S. Modha. 2001. Concept Decompositions forLarge Sparse Text Data Using Clustering.
Machine Learning
42 (2001), 143–175.[11] Yuxiao Dong, Hao Ma, Zhihong Shen, and Kuansan Wang. 2017. A Century ofScience: Globalization of Scientific Collaborations, Citations, and Innovations.In
Proceedings of the 23rd ACM SIGKDD International Conference on KnowledgeDiscovery and Data Mining . 1437–1446.[12] Santo Fortunato, Carl T. Bergstrom, Katy Borner, James A. Evans, Dirk Helbing,Stasa Milojevic, Alexander M. Petersen, Filippo Radicchi, Roberta Sinatra, BrianUzzi, Alessandro Vespignani, Ludo Waltman, Dashun Wang, and Albert LaszloBarabasi. 2018. Science of science.
Science
ICETiCfi09 : International Conference on EmergingTrends in Computing (2009), 309–315.[14] Khalid Haruna, Maizatul Akmar Ismail, Abdullahi Baffa Bichi, Victor Chang,Sutrisna Wibawa, and Tutut Herawan. 2018. A Citation-Based RecommenderSystem for Scholarly Paper Recommendation.. In
International Conference onComputational Science and Its Applications . 514–525.[15] Kurt Hornik, Ingo Feinerer, and Martin Kober. 2012. Spherical k-Means Cluster-ing.
Journal of Statistical Software
50, 1 (2012), 1–22.[16] Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuningfor text classification. In
Proceedings of the 56th Annual Meeting of the Associationfor Computational Linguistics (Volume 1: Long Papers) , Vol. 1. 328–339.[17] Wenyi Huang, Zhaohui Wu, Prasenjit Mitra, and C Lee Giles. 2014. Refseer: Acitation recommendation system. In
Digital Libraries (JCDL), 2014 IEEE/ACMJoint Conference on
Proceedings of the third ACMconference on Recommender systems . ACM, 241–244.[20] Quoc V. Le and Tomas Mikolov. 2014. Distributed Representations of Sentencesand Documents. international conference on machine learning (2014), 1188–1196.[21] Maake Benard Magara, Sunday O. Ojo, and Tranos Zuva. 2018. A comparativeanalysis of text similarity measures and algorithms in research paper recom-mender systems. In .[22] Tomas Mikolov, Kai Chen, Greg S. Corrado, and Jeffrey Dean. 2013. Efficient Esti-mation of Word Representations in Vector Space. arXiv preprint arXiv:1301.3781 (2013).[23] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.Distributed representations of words and phrases and their compositionality. In
Advances in neural information processing systems . 3111–3119.[24] Cristiano Nascimento, Alberto H.F. Laender, Altigran S. da Silva, and Mar-cos Andr Gonalves. 2011. A source independent framework for research paperrecommendation. ACM Press, 297.[25] Zhihong Shen, Hao Ma, and Kuansan Wang. 2018. A web-scale system forscientific knowledge exploration. meeting of the association for computationallinguistics (2018).[26] Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Paul Hsu,and Kuansan Wang. 2015. An Overview of Microsoft Academic Service (MAS)and Applications. In
Proceedings of the 24th International Conference on WorldWide Web . 243–246.[27] Henry Small. 1973. Co-citation in the scientific literature: A new measure ofthe relationship between two documents.
Journal of The American Society forInformation Science
24, 4 (July 1973), 265–269.[28] Roberto Torres, Sean M. McNee, Mara Abel, Joseph A. Konstan, and John Riedl.2004. Enhancing digital libraries with TechLens+. In
Proceedings of the 4thACM/IEEE-CS joint conference on Digital libraries . ACM, 228–236.. ACM, 228–236.