Community Detection and Growth Potential Prediction Using the Stochastic Block Model and the Long Short-term Memory from Patent Citation Networks
Kensei Nakai, Hirofumi Nonaka, Asahi Hentona, Yuki Kanai, Takeshi Sakumoto, Shotaro Kataoka, Elisa Claire Alemán Carreón, Toru Hiraoka
Abstract - S coring patent documents is very useful for technology management. However, conventional methods are based on static models and, thus, do not reflect the growth potential of the technology cluster of the patent. Because even if the cluster of a patent has no hope of growing, we recognize the patent is important if PageRank or other ranking score is high. Therefore, there arises a necessity of developing citation network clustering and prediction of future citations. In our research, clustering of patent citation networks by Stochastic Block Model was done with the aim of enabling corporate managers and investors to evaluate the scale and life cycle of technology. As a result, we confirmed nested SBM is appropriate for graph clustering of patent citation networks. Also, a high MAPE value was obtained and the direction accuracy achieved a value greater than 50% when predicting growth potential for each cluster by using LSTM.
Keywords – Patent, Stochastic Block Model, Long Short-Term Memory, Patent scoring
I. INTRODUCTION
Patents can protect the product innovation of firms from rivals and create a financial benefit [1]. Therefore, patent information is widely used for many studies of technology management such as analysis of technology driven industry [2-3] and real option theory [4]. Patent data is also useful to identify technology trends which help managers plan strategies of R&D, M&A, and other business process. Some researchers applied text-mining techniques to capture technology trends [5-8]. For examples, Nonaka et al, [8] proposed a method for extracting the effect and the technological information from patents by using entropy-based boot-strap and grammar patterns. While information extraction of patent documents is important to analyze technology, the scoring of patents is also useful for technology management analysis. There are many aspects of patent scoring such as patent application number based index [9]. Among them, the number of cited patents is especially used for quantification of patent value because it shows a degree of attention from other competitors in the same technology field. Hall [10] found that patent citation significantly affects market value. Chang [11] study employs a panel threshold regression model to test whether the patent i-index has a threshold effect on the relationship between patent citations and market value in the pharmaceutical industry. However, simple citation counting, or its expansion, should not be considered the inherent value of importance of different patents. Therefore, it is necessary to assign importance ranks to each patent. To address the issue, a link evaluation method such as PageRank [12] or HITS [13] has been used for calculating importance of patents. Lukach, et al. [14] have proposed computing importance by the PageRank score of patents. According to their study, PageRank patent importance weights differs from the weights based on the number of backward or forward citations. Nonaka, et al. [15] calculated a correlation between the patent score based on HITS, which is similar with Pagerank score, and stock data. They indicated high correlation between the patent score of B to B companies and their time series of stock price value. Additionally, Bruck [16] showed that using the PageRank score of patent documents gives the possibility to make predictions about future technological development. While link evaluation methods are useful for patent scoring, these methods are based on static models and thus do not reflect the growth potential of the technology cluster of the patent. Because, even if the cluster of a patent has no hope of growing, we recognize the patent is important if PageRank or other ranking score is high. Therefore, there arises a necessity of developing citation network clustering and prediction of future citations. In this paper, we proposed a community detection method using the Stochastic Block Model (SBM) [17] and an evaluation of growth potential method using the Long Short-Term Memory (LSTM) [18] deep learning model. SBM is a generative model for random graphs to group similar nodes to communities. In this research, we adopted a Nested SBM [19] which is suitable for large networks into citation networks. Then we predicted the future citation number of each technology communities using the Long Short Time Memory deep learning model which is widely used for time series data such as financial data [20]. The remainder of this paper is organized as follows. In section 2, we describe the method which is combined between nested SBM as community detection from patent citation network, and LSTM as patent communities' growth
Community Detection and Growth Potential Prediction Using the Stochastic Block Model and the Long Short-Term Memory from Patent Citation Networks
Kensei Nakai , Hirofumi Nonaka , Asahi Hentona , Yuki Kanai , Takeshi Sakumoto , Shotaro Kataoka , Elisa Claire Alemán Carreón , Toru Hiraoka Department of Information and Management Systems Engineering, Nagaoka University of Technology, Nagaoka, Japan ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]) Department of Information Systems, University of Nagasaki, Nagasaki, Japan ([email protected]) analysis. Our experiments on system performance and the result of the analysis are explained in section 3. Section 4 conducts a discussion and evaluation of this system based on the experiment results. Finally, section 5 describes the conclusion of the paper and our future work. II. METHODOLOGY A. Overview of Our method
Our proposed method is shown in figure 1.
Figure 1
Outline of our method B. Citation network graph extraction from patent database
Using the patent information database as an input, we created a graph file of patent citation network in patent field to be analyzed. First, we prepared a patent information database. In this study, we used a Japanese patent database provided by the Institute of Intellectual Property of Japan (IIP) [21]. It contains about 6,000,000 patents from 1960 to 2013 and their examiner citations. After that, it extracts the patent citations network in the patent field to be analyzed from the database and generates a graph file (DOT file). In the graph file, the nodes of the network (patent application number) and the link (quotation relation of the patent) are recorded. This generated graph file is used for input of nested SBM to be performed later. C. Community detection using nested SBM
In this paper, we use Nested SBM [19] to detect patent community. The original SBM classifies nodes into blocks (communities) with edges between each nodes of blocks. The overview of SBM is shown in Figure 2. Nested SBM is a hierarchy SBM model and can detect the best partition Graph-tool https://graph-tool.skewed.de/
Figure 2 Overview of SBM number automatically. For more details, please refer to reference [19]. D. Growth analysis and prediction of communities using LSTM
In this research, we used long short-term memory (LSTM) networks, one of the most advanced deep learning architectures. This model is used in sequence learning tasks such as character recognition, or time series prediction [22-23] to predict citation number of each community. For more details, please refer to reference [20]. III. RESULTS A. Nested-SBM
In this section, we applied graph clustering algorithm to real patent networks. We analyzed the citation network of the game technology field including "Video games and related technology fields" (IPC code: A63F13) and "Roulette-like ball games and related technology fields" (IPC code: A63F5) for its convenience for understanding the technology and evaluating results. Table I shows the details of the data. In order to conduct the Nested SBM algorithm, we used Python library graph-tool developed by Tiago de Paula Peixoto, and clustered patent citation networks. Figure 3 shows the clustering result by “nested SBM”. (a) “Video games” (Max. clusters: 36) (b) “Roulette-like ball games” (Max. clusters: 88) Figure 3 Result of Clustering (Expressed as graphical)
This figure is a graphical representation of the result of the hierarchical clustering. The cluster is integrated inside the figure as the bottom layer clusters (number of clusters K = target patent number) on the outside of the figure, and it finally shows that it connects to one cluster. B. LSTM
We predicted the number of citations of each community to evaluate the growth potential of the community by LSTM. We used Python library
TensorFlow . Our LSTM Neural Network is composed of one input layer, four LSTM layers with memory blocks, and one output layer. We computed the Mean Absolute Percentage Error (MAPE) and accuracy of direction, to measure the effectiveness of different travel speed algorithms. Table II shows the system performance of our LSTM, and Figure 4 and figure 5 show prediction results of patent citations using LSTM. The horizontal axis of each graph is the patent application year (the origin of the target data is 1960 as the origin of the horizontal axis), and the vertical axis is the number of citations of the patent. The red line of each graph is true value, the green line is prediction by training data, and the blue line is prediction by test data. Table II System Performance
MAPE (%) Accuracy of direction (%) Video games 62.06 53.94 Roulette-like ball games 70.82 57.1
Figure 4 Result of Prediction (bad example)
Table I
Network to Be Analyzed
Patent classification name Number of patent nodes Number of cited links Video games 15631 26826 Roulette-like ball games 29621 74529
Figure 5 Result of Prediction (good example)
IV. DISCUSSION A. Nested SBM
Using the results of Nested SBM, we examined the technical field by randomly referring to patent documents of patents belonging to each cluster. As a result, we could find common technical points. Regarding the technology classification "Video Game and related technology fields", 534 patents were applicable to a certain cluster.
This cluster is a set of patents of the technique of the game user interface and its technology. The central patent (JP 2003-340042) is a patent concerning "a pachinko machine having a function to convey a special event state in the game in an easy-to-understand manner".
This patent affects other patents on the way a user interacts with the game. In addition, this cluster includes patents of "screen drawing optimization" and "home game controller". In another cluster, 340 patents were applicable. The cluster is a collection of patents related to input and output of game machines and control of peripheral devices. The central patent is a patent on a music game system and a play screen structure (JP 平 pachinko / pachislot machines and their peripheral technology. The central patent is a circuit board box (JP 平 B. Long Short-Term Memory
As a result of prediction by LSTM, the MAPE value was not good. In particular, it was revealed that the MAPE value was bad in a community that rapidly declined after the patent citation rapidly increased. On the other hand, we can predict patent citations well for communities with long-term trends. According to Table II, prediction of direction showed more than 0.5 accuracy in any technical classification, but there is still a room for improvement. In the future, it is necessary to consider the number of nodes and the number of layers of LSTM for improvement. V. CONCLUSION In our research, clustering of patent citation networks by Stochastic Block Model was done with the aim of enabling corporate managers and investors to evaluate the scale and life cycle of technology. As a result, we confirmed nested SBM is appropriate for graph clustering of patent citation networks and the not good MAPE value and direction accuracy more than 0.5 of predicting growth potential of each cluster by using LSTM. In future work, we improve LSTM to predict growth potential of each cluster. Furthermore, we combined our proposed method and a link evaluation method for investors and corporate managers to predict corporate finance data. REFERENCES [1] P. H. Sullivan, “Profiting from intellectual capital”, Journal of Knowledge Management, Vol. 3 Issue: 2, pp.132-143, 1999. [2] M. Stolpe, “Determinants of knowledge diffusion as evidenced in patent data: the case of liquid crystal display technology”, Research Policy 31.7, 2002, pp. 1181-1198. [3] Joel M. Podolny, T. E. Stuart, and M. T. Hannan. “Networks, knowledge, and niches: Competition in the worldwide semiconductor industry, 1984-1991.”, American journal of sociology 102.3, 1996, pp. 659-689. [4] A. C. Marco, “The option value of patent litigation: Theory and evidence.” Review of Financial Economics 14.3-4, 2005, pp. 323-351. [5] Y. Suzuki, H. Nonaka, H. Sakaji, A. Kobayashi, H. Sakai and S. Masuyama, “NTCIR-8 Patent Mining Task at Toyohashi University of Technology”, NTCIR, pp. 364-369, 2010. [6] H. Sakaji, H. Nonaka, H. Sakai, and S. Masuyama, “Cross-Bootstrapping: An Automatic Extraction Method of Solution-Effect Expressions from Patent Documents”, IEICE D, vol. 93, no.6, pp. 742-755, 2010. [7] A. Kobayashi, H. Nonaka, H. Sakai, and S. Masuyama, “An automatic thesaurus construction method for technological terms in patent maps”, Computers and Industrial Engineering (CIE), 2010 40th International Conference on. IEEE, 2010. [8] H. Nonaka, A. Kobayashi, H. Sakaji, Y. Suzuki, H. Sakai, S. Masuyama, “Extraction of the Effect and the Technology Terms from a Patent Document”, Journal of Japan Industrial Management Association, vol.63, pp.105 - 111, 2012. [9] Y. S. Chen, and K. C. Chang, “The relationship between a firm's patent quality and its market value - the case of US pharmaceutical industry”, Technological Forecasting and Social Change, vol.77, no.1, pp. 20-33, 2010. [10] B. H. Hall, A. Jaffe, M. Trajtenberg. “Market value and patent citations”, RAND Journal of economics, pp. 16-38, 2005. [11] K. C. Chang, W. Zhou, S. Zhang, C. C. Yuan “Threshold effects of the patent H-index in the relationship between patent citations and market value”, Journal of the Association for Information Science and Technology, vol.66, no.12, pp. 2697-2703, 2015. [12] L. Page, S. Brin, R. Motwani, T. Winograd, “The PageRank citation ranking: Bringing order to the web”, Stanford InfoLab, 1999. [13] J. Kleinberg, “Authoritative sources in a hyperlinked environment”, Journal of the ACM, Vol. 46, issue 5, pp. 604-632, 1999. [14] R. Lukach, and M. Lukach. “Ranking USPTO patent documents by importance using random surfer method (PageRank)”, Available at SSRN: http://ssrn.com/abstract=996595, 2007. [15] H. Nonaka, D. Kubo, M. Kimura, T. Hiraoka, T. Ota and S. Masuyama. “Correlation analysis between financial data and patent score based on HITS algorithm”, 2014 IEEE International Technology Management Conference, 2014. [16] P. Bruck, I. Réthy, J. Szente, J. Tobochnik, P. Érdi, “Recognition of emerging technology trends: class-selective study of citations in the US Patent Citation Network”, Scientometrics, vol. 107, issue 3 pp.1465-1475, 2016. [17] P. W. Holland, K. B. Laskey, S. Leinhardt, “Stochastic blockmodels: First steps”, Social networks, vol. 5, issue 2, pp. 109-137, 1983. [18] S. Hochreiter and J. Schmidhuber, “Long short-term memory”, Neural computation, vol.9, issue 8, pp. 1735-1780, 1997. [19] T. P. Peixoto, “Hierarchical block structures and high-resolution model selection in large networks”, Phys. Rev. X 4, 011047, 2014. [20] T. Fischer and C. Krauss, “Deep learning with long short-term memory networks for financial market predictions." European Journal of Operational Research, 2017. [21] A. Goto, K. Motohashi, “Construction of a Japanese Patent Database and a First Look at Japanese Patenting Activities”, Research Policy, vol. 36 issue 9, pp. 1431-1442, [22] A.Graves, M. Liwicki, S. Fernández, R. Bertolami, H. Bunke and J. Schmidhuber, “A novel connectionist system for unconstrained handwriting recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.31, No.5, pp.855-868, 2009. [23] A. Graves, A.R. Mohamed and G. Hinton, “Speech recognition with deep recurrent neural net-works”, In Proceedings of the 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649, 2013.[22] A.Graves, M. Liwicki, S. Fernández, R. Bertolami, H. Bunke and J. Schmidhuber, “A novel connectionist system for unconstrained handwriting recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.31, No.5, pp.855-868, 2009. [23] A. Graves, A.R. Mohamed and G. Hinton, “Speech recognition with deep recurrent neural net-works”, In Proceedings of the 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649, 2013.